Skip to content
Book Open access

ADROIT: Adaptive Decision for Reasoning-Optimized Interleaved Thought in Multimodal Reasoning

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 29 references

Abstract

Interleaved-modal Chain-of-Thought (I-MCoT) has enhanced the capabilities of Vision-Language Models (VLMs) in Visual Question Answering (VQA) by grounding the reasoning process in inserted visual evidence. However, existing approaches often operate under a largely unexamined assumption that visual insertions are advantageous. In contrast, our analysis reveals a fundamental trade-off: while the introduced visual information aids reasoning, the insertion action could impair reasoning fluency and model performance, with the impairment scaling with token quantity and insertion frequency. Ultimately, the detrimental effects can outweigh the benefits of visual insertion. In this paper, we propose ADROIT, a training-free I-MCoT framework that explicitly models and resolves this trade-off via two synergistic components: (1) Visual Sufficiency Trigger (VST), which helps a VLM avoid superfluous insertions by identifying insertion moments where the gain from visual information is theoretically guaranteed. (2) Optimal Transport Planner (OTP), which retrieves the compact, most reasoning-relevant, and semantically coherent visual evidence for a VLM by formulating evidence selection as an entropy-regularized optimal transport problem. Extensive experiments on five benchmarks and four backbones confirm the consistent superiority of our proposed approach. Our code is available at https://github.com/HsipingLi/ADROIT.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.