Skip to content

SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection

Oct 2026 · 0 citations · 24 references
Computer Science

TL;DR

The key innovation of SOPO is a Socratic feedback mechanism, where a teacher model guides the reasoning trajectories during rollout by providing indirect hints inferred from the ground truth, enabling the model to generate more effective reasoning trajectories for hard examples, thus alleviating reward sparsity.

Abstract

Large Language Models (LLMs) have gained widespread adoption in various natural language processing tasks, but suffer from hallucination issues where they generate unfaithful or inconsistent content. While Reinforcement Learning with Verifiable Rewards (RLVR) has shown promising applications in reasoning tasks, its application to span-level hallucination detection is challenged by localization ambiguity, reward sparsity on hard examples, and uncontrolled length inflation. To address these issues, we introduce Socratic Guided Policy Optimization (SOPO), a multi-turn RL framework for robust hallucination detection. The key innovation of SOPO is a Socratic feedback mechanism, where a teacher model guides the reasoning trajectories during rollout by providing indirect hints inferred from the ground truth. This teacher-guided approach enables the model to generate more effective reasoning trajectories for hard examples, thus alleviating reward sparsity. Furthermore, we incorporate Turn-based Advantage Scaling to penalize verbosity bias, encouraging the model to develop efficient and autonomous reasoning capabilities. Empirical results on RAGTruth demonstrate that SOPO (4B) achieves state-of-the-art performance, surpassing leading proprietary models (e.g., GPT-5) and standard RL baselines, while obtaining higher reported mean scores than GRPO on four out-of-domain benchmarks, with mixed evidence against CAPO. We release our training and evaluation code alongside the 1.7B and 4B SOPO models at https://github.com/patsnap/SOPO.

View source

Similar papers

Alleviating Hallucination in Reasoning Tasks with Training-Free Uncertainty-Guided Steering

USteer, a simple, training-free steering mechanism that adjusts a model's layer-wise activations during inference using the gradient of a confidence measure with respect to the activations, demonstrates that confidence signals can be leveraged not only for detection, but also for effective inference-time control of mod...

Li-Tian Liu, Qi-Qi Hou, Yu-Bin Jian et al. · 0 citations
Preprint Nov 2026

From Twisted Facts to Rogue Agents: Can LLMs Reliably Detect Multi-Dimensional Hallucinations?

Large language models (LLMs) are widely used for information access, content production, and agent-based task execution, but their outputs remain vulnerable to subtle, high-risk hallucinations. To systematically evaluate and advance post-generation hallucination detection, we introduce HaluCheck, a large-scale, bilingu...

Han-Yu Qiu, Ze-Li Liu, Zhe Zhao et al. · 0 citations

DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs

Dual-Entropy Enhanced Policy Optimization (DEEPO), a dual-stage enhancement combining signal variance regularization with gradient preconditioning with semantic-entropy-triggered expert prefixes inject grounded continuations on high-uncertainty queries, providing direct supervision and restoring advantage variance.

Ying-Xuan Zhuang, Miao Pan, Wang-Jie Gan et al. · 0 citations
Book Open access Nov 2026

Mitigating Hallucinations in Large Multimodal Reasoning Models by Regulating Parametric Prior Drift

Large Multimodal Reasoning Models (LMRMs) have recently shown strong capabilities in complex visual understanding and multi-step inference. However, they remain highly vulnerable to multimodal hallucinations over extended reasoning trajectories. In this work, we study this failure as a state-dynamics problem: as reason...

Xiao-Ye Qu, Yan-Zhe Wu, Zhi-Hao Xie et al. · 0 citations
Preprint Nov 2026

Entropogram-guided Visual Realignment Decoding: Capturing the Rhythm of Multimodal Reasoning

This work revisit hallucination in MLLMs from the perspective of reasoning generation dynamics and proposes an entropy-aware self-introspective framework for MLLM reasoning, arguing that entropy is not merely a static measure of uncertainty but a structured signal that reflects how modality information is integrated th...

Jia-Qi Deng, Zhe Fu, Ya-Kun Chen et al. · 0 citations
Preprint Nov 2026

MIRA: Reinforcing Multimodal Reasoning via Deceptive Contextual Augmentation

While Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced multimodal reasoning, existing frameworks suffer from a critical"perception-reasoning gap."Due to an over-reliance on seemingly relevant visual tokens, even minor perceptual perturbations can propagate through the reasoning process,...

Zhi-Han Yin, Jian-Xin Liang, Yi-Feng Yao et al. · 1 citation

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.