A new multimodal ICL framework is proposed that combines contrastive demonstration modeling with the self-refinement capability of MLLMs and consistently improves MLLM performance, with particularly notable gains on visual question answering (VQA).
Abstract
In-context learning (ICL) is widely used in multimodal large language models (MLLMs) and achieves strong performance across a wide range of multimodal tasks. However, existing multimodal ICL methods often rely on surface level imitation of in-context demonstrations, making it difficult for MLLMs to align their responses with the reasoning path required by the given multimodal input. This limitation becomes more pronounced in complex multimodal tasks, thereby restricting further improvements in MLLM performance. To address this issue, we propose a new multimodal ICL framework that combines contrastive demonstration modeling with the self-refinement capability of MLLMs. Specifically, our framework reformulates each demonstration by explicitly contrasting a suboptimal response with a better response under the same input, together with a reasoning path that reveals how the response should be refined. This contrastive formulation makes the reasoning path toward the desired response more explicit and guides the MLLM beyond superficial imitation. Furthermore, because effective refinement depends on the current response, we introduce a response-conditioned retrieval mechanism to select demonstrations whose reasoning paths are more relevant to the current response. In addition, we use a lightweight alignment controller to predict response quality and determine whether further refinement is needed. Experiments on three types of multimodal tasks show that the proposed framework consistently improves MLLM performance, with particularly notable gains on visual question answering (VQA).
Large vision-language models (LVLMs) exhibit strong multimodal in-context learning (ICL) capabilities, yet this ability degrades substantially as model size decreases. Knowledge distillation offers a natural way to bridge this gap, but existing methods primarily align output distributions or hidden representations dire...
MMDynOpt-Agent is a lightweight multimodal agent that models the dynamic optimization of multimodal reasoning as a Markov decision process via end-to-end reinforcement learning, designed to reduce the cost of multimodal reasoning.
Wen-Jin Liu, Hao-Ran Luo, Fa-Yuan Ke et al.· 0 citations
Func-R1 is proposed, which synergistically harmonizes precise visual perception and rigorous logical reasoning and delivers the optimal performance among open-source MLLMs, even surpassing GPT-5 with an 8.4% improvement on MathVerse's function-oriented tasks.
Ming Yin, Xiaohai Wang, Dian Li et al.· Conference on Empirical Meth...· 1 citation
V-ICAL is introduced, a novel benchmark designed to evaluate video-based ICL for multimodal agents that seamlessly connects in-context knowledge induction with core agentic capabilities, including state grounding, temporal memory, planning, and adaptation in dynamic environments.
Zi-Qian Fan, Shi-Bo Xu, Jun-Jie Li et al.· 0 citations
Recent Omni-Modal Generative Models (Omni-Models) have advanced content generation toward unified modeling of text, images, video, and audio. MiniMax-H3 exemplifies this transition by combining multimodal context understanding with joint audio-visual generation in a shared latent framework. Its unified architecture rai...
Hao-Yu Zhao, Zi-Hao Zhao, Tian-Yuan Deng et al.· 1 citation
Active multimodal agents use visual tools to acquire task-relevant evidence while reasoning. Although reinforcement learning samples multiple interaction trajectories per input, outcome-based objectives primarily use the group to estimate scalar advantages, leaving complementary visual discoveries underused. We introdu...
Zheng Jiang, Hou-De Qian, Yi-Ming Chen et al.· 0 citations