Empirical Analysis of Chain-of-Thought and Solver-Augmented Large Language Models for Deductive Reasoning

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in multistep reasoning, yet fundamental questions remain about their ability to perform reliable deductive reasoning. Two contrasting paradigms have emerged: chain-of-thought (CoT) prompting, which treats LLMs as self-contained reasoners, and solver-augmented approaches, which use LLMs as semantic parsers to translate problems into formal logic for symbolic execution. We present a systematic comparison of these approaches under strict single-pass evaluation across three established benchmarks (ProofWriter, PrOntoQA, and LogicalDeduction) and controlled synthetic datasets. Our findings reveal three key insights: (1) on standard benchmarks, solver augmentation yields minimal improvements ( < 1 % ) over zero-shot CoT for state-of-the-art models, suggesting diminishing performance margins between paradigms under current benchmark settings; (2) mid-sized open-source models (14B–32B) match proprietary model performance when solver-augmented, with the primary challenge shifting from generating valid syntax (small models) to accurate semantic parsing (larger models); and (3) on synthetic datasets with controlled difficulty variations, solver-augmented methods demonstrate superior robustness, maintaining 76.7% accuracy retention at extreme reasoning depths (14–17 steps) compared to 62.2% for CoT, and showing greater resilience to distracting facts and rules. These results indicate that while CoT-augmented LLMs achieve strong performance on deductive reasoning tasks up to five hops, solver augmentation remains valuable for deeper multihop deduction and for applications requiring robust and verifiable reasoning.