Difficulty-aware Dynamic Chain-of-Thought Prompting for Large Language Models via BM25 and Semantic Retrieval
TL;DR
A dynamic Chain-of-Thought prompting method based on problem difficulty assessment that effectively resolve the two major limitations of traditional exemplar-based methods, enabling LLMs to obtain appropriately tailored exemplars for both multi-step mathematical reasoning and interdisciplinary question answering.
Abstract
Large language models (LLMs) have demonstrated exceptional capabilities across multiple domains and emerged as the core driving force in natural language processing. Their reasoning however can be associated with logical flaws and lack of stability when attempting to solve difficult problems that require multi-step deduction, cross-domain knowledge or implicit constraints, with redundant or insufficient exemplars in conventional prompts and poor fit with exemplar and target problems. To address these issues, we propose a dynamic Chain-of-Thought (CoT) prompting method based on problem difficulty assessment: first, the model performs zero-shot self-evaluation of the required solution steps to dynamically determine the number of exemplars; then, it integrates BM25 retrieval to select the most similar high-quality question-answer pairs, constructing precise Few-shot prompts. Experiments conducted on multiple datasets effectively resolve the two major limitations of traditional exemplar-based methods, enabling LLMs to obtain appropriately tailored exemplars for both multi-step mathematical reasoning and interdisciplinary question answering. Consequently, the accuracy of complex reasoning is improved to varying degrees across tasks.