INSPIRE: An Internalize-Then-Improve Approach for Example-Driven Mathematical Reasoning

Find Tailored Step Example for Next Step: a Targeted Step-wise Retrieval Framework for Guiding LLM Reasoning

Large language models (LLMs) have shown strong performance in mathematical reasoning, supported by approaches such as In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG). However, existing methods often provide problem-level examples, which is too coarse-grained for multi-step reasoning to cause informational redundancy, and structural misalignment. To address this limitation, we propose Step-wise Training for In-context Reasoning (STIR) to provide step-synchronized and logically targeted guidance to enhance the model's mathematical reasoning capabilities. STIR enables a model to dynamically decide when to retrieve a single logically consistent next step, just using the current problem and its intermediate state as the query. First, We decompose expert solutions into Step-Level Reasoning Units inspired by human thinking patterns. Leveraging this data, a Step Retriever is trained for logical continuity to map current reasoning states to relevant subsequent steps. Then a Step Reasoner is trained to decide when to retrieve tailored step examples and incorporates this guidance into reasoning. We further extend STIR with a Process-aware Reinforcement Learning phase using Group Relative Policy Optimization to learn to self-formulate search queries and optimizes the decision-making policy. Experiments on seven benchmarks demonstrate that STIR achieves accuracy improvements ranging from 1.86% to 17.96%, maintaining lower token efficiency than baselines. Analysis via our proposed DSM, TCN and RCR metrics shows that STIR improves reasoning capability, achieving DSM scores ranging from 4.54 to 29.87 across backbones and significant improvements over the baseline in both TCN and RCR.

Assessing LLMs'mathematical abilities requires understanding the various mechanisms of mathematical creativity

How should we assess whether large language models can perform mathematical invention? I argue that this question is currently underspecified: mathematical creativity is not one capacity but several mechanistically distinct modes of meaning-making - reflexive introspection on mathematical practice, analogical import from the sciences, problem-driven construction, and the bridging of distant domains - together with a further, cross-cutting distinction between meaning pursued because a pattern was observed and meaning pursued because it is strategically wanted, a distinction I develop through the case of conjecture-formation. These mechanisms are likely non-substitutable, so that competence in one does not transfer to the others. Grounding each in a historical case study and in an architecture-level account of current transformer-based systems, I suggest that today's models concentrate their competence in modes shaped by recombination and search over existing building blocks; if that description holds, the remaining modes are out of reach in principle, not just slower - though whether it holds is itself the open, empirical part. Because proof is getting cheaper as AI improves at generating it - a shift the field's own leading voices are now diagnosing - mathematical value is migrating toward the modes current systems cannot yet perform, and evaluations of AI mathematical ability should be organized around this taxonomy rather than around aggregate benchmarks that conflate it.

Activation Steering for Enhancing Mathematical Reasoning in Large Language Models: A Study of Cross-Model Generalization

An activation steering method based on single-vector ablation is proposed to enhance mathematical reasoning by injecting a carefully constructed steering vector into the model’s residual stream by constructing the AS direction from the activation difference between mathematical and general-domain samples.

Meta-Cognitive Judgment

ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling

ThinkRetrieve is proposed, a test-time scaling framework that augments the reasoning traces of LRMs with dynamically retrieved solved examples at each reasoning step, providing the model with guidance on how to reason rather than merely what facts are relevant.

Magenta: Closing the Loop Between Mathematical Reasoning and Lean Verification

Most of mathematical knowledge has been communicated through so-called informal use of mathematics and natural language. With large language models (LLMs) being highly adept in using natural language, they achieve strong performance, yet not perfect, in informal mathematical reasoning. Restraining LLMs to informal reasoning misses out on the opportunity to use the discrete verification abilities that machines offer through machine-checkable proofs. In this paper, we bridge the gap between informal and formal reasoning by integrating Lean signals into the informal reasoning process. We introduce Magenta, a training-free agentic pipeline that, given only a natural-language problem, produces an answer, expresses it as a Lean 4 statement, and constructs a machine-checked proof. A statement judge verifies whether the formalisation preserves the original problem, while an error-attribution judge routes failed attempts either to mathematical re-derivation or local Lean repair. Magenta achieves 100% accuracy across all evaluated olympiad benchmarks, including AIME 2025, AIME 2026, and HMMT February 2026. When paired with the open-weight K2-Horizon-7B reasoner, it solves all six IMO 2026 problems. Our analysis shows that statement adjudication is essential for preventing false certificates and that feedback-guided correction outperforms independent resampling on difficult problems.