Activation Steering for Enhancing Mathematical Reasoning in Large Language Models: A Study of Cross-Model Generalization

Abstract

Large Language Models (LLMs) often exhibit limited performance on mathematical reasoning tasks. This paper proposes an activation steering (AS) method based on single-vector ablation to enhance mathematical reasoning by injecting a carefully constructed steering vector into the model’s residual stream. Specifically, the AS direction is constructed from the activation difference between mathematical and general-domain samples and is injected into designated transformer layers during inference. Experiments on Llama-3-8B-Instruct demonstrate that the proposed method improves mathematical reasoning accuracy from 11.0% to 39.6%, while simultaneously enhancing general capabilities as well. However, transfer experiments on Qwen2.5-7B-Instruct fail to achieve comparable improvements, revealing the strong model dependency of the proposed approach. Furthermore, the effects of layer selection, steering coefficient, positional window, and steering vector construction are systematically investigated. This study provides both empirical evidence and theoretical insights regarding the application of activation steering methods.