The key innovation of SOPO is a Socratic feedback mechanism, where a teacher model guides the reasoning trajectories during rollout by providing indirect hints inferred from the ground truth, enabling the model to generate more effective reasoning trajectories for hard examples, thus alleviating reward sparsity.
Abstract
Large Language Models (LLMs) have gained widespread adoption in various natural language processing tasks, but suffer from hallucination issues where they generate unfaithful or inconsistent content. While Reinforcement Learning with Verifiable Rewards (RLVR) has shown promising applications in reasoning tasks, its application to span-level hallucination detection is challenged by localization ambiguity, reward sparsity on hard examples, and uncontrolled length inflation. To address these issues, we introduce Socratic Guided Policy Optimization (SOPO), a multi-turn RL framework for robust hallucination detection. The key innovation of SOPO is a Socratic feedback mechanism, where a teacher model guides the reasoning trajectories during rollout by providing indirect hints inferred from the ground truth. This teacher-guided approach enables the model to generate more effective reasoning trajectories for hard examples, thus alleviating reward sparsity. Furthermore, we incorporate Turn-based Advantage Scaling to penalize verbosity bias, encouraging the model to develop efficient and autonomous reasoning capabilities. Empirical results on RAGTruth demonstrate that SOPO (4B) achieves state-of-the-art performance, surpassing leading proprietary models (e.g., GPT-5) and standard RL baselines, while obtaining higher reported mean scores than GRPO on four out-of-domain benchmarks, with mixed evidence against CAPO. We release our training and evaluation code alongside the 1.7B and 4B SOPO models at https://github.com/patsnap/SOPO.
USteer, a simple, training-free steering mechanism that adjusts a model's layer-wise activations during inference using the gradient of a confidence measure with respect to the activations, demonstrates that confidence signals can be leveraged not only for detection, but also for effective inference-time control of mod...
Large language models (LLMs) are widely used for information access, content production, and agent-based task execution, but their outputs remain vulnerable to subtle, high-risk hallucinations. To systematically evaluate and advance post-generation hallucination detection, we introduce HaluCheck, a large-scale, bilingu...
Han-Yu Qiu, Ze-Li Liu, Zhe Zhao et al.· 0 citations
Dual-Entropy Enhanced Policy Optimization (DEEPO), a dual-stage enhancement combining signal variance regularization with gradient preconditioning with semantic-entropy-triggered expert prefixes inject grounded continuations on high-uncertainty queries, providing direct supervision and restoring advantage variance.
Ying-Xuan Zhuang, Miao Pan, Wang-Jie Gan et al.· arXiv.org· 0 citations
Large Multimodal Reasoning Models (LMRMs) have recently shown strong capabilities in complex visual understanding and multi-step inference. However, they remain highly vulnerable to multimodal hallucinations over extended reasoning trajectories. In this work, we study this failure as a state-dynamics problem: as reason...
Xiao-Ye Qu, Yan-Zhe Wu, Zhi-Hao Xie et al.· Proceedings of the 34th ACM...· 0 citations
This work revisit hallucination in MLLMs from the perspective of reasoning generation dynamics and proposes an entropy-aware self-introspective framework for MLLM reasoning, arguing that entropy is not merely a static measure of uncertainty but a structured signal that reflects how modality information is integrated th...
Jia-Qi Deng, Zhe Fu, Ya-Kun Chen et al.· 0 citations
While Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced multimodal reasoning, existing frameworks suffer from a critical"perception-reasoning gap."Due to an over-reliance on seemingly relevant visual tokens, even minor perceptual perturbations can propagate through the reasoning process,...
Zhi-Han Yin, Jian-Xin Liang, Yi-Feng Yao et al.· 1 citation