Skip to content

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

Jun 2026 · arXiv.org · Vol abs/2606.02502 · 0 citations · 67 references
Computer Science

TL;DR

By isolating task-specific patterns into independent modules, CRAM mitigates catastrophic forgetting across tasks and boost parameter efficiency, and utilizes adaptive-rank instantiation to identify the capability gap between existing expert capability and new task demands, and dynamically allocate only the necessary parameters.

Abstract

Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Continual Instruction Tuning (MCIT) essential. Existing methods either update all tasks with a shared parameter set or allocate dedicated modules for each new task. Shared updates force heterogeneous tasks to compete, causing forgetting of learned capabilities. Conversely, isolated expansion prevents interference but severely limits parameter efficiency over long task streams. To address this dilemma, we propose CRAM (Centroid-Routing and Adaptive MoE). Specifically, by isolating task-specific patterns into independent modules, CRAM mitigates catastrophic forgetting across tasks. To further boost parameter efficiency, we utilize adaptive-rank instantiation to identify the capability gap between existing expert capability and new task demands, and dynamically allocate only the necessary parameters. To ensure stable reuse among tasks, centroid-guided routing recognizes and activates existing experts'capabilities, while an orthogonality penalty confines new updates to task-specific directions, preventing re-learning general capability. Extensive experiments across diverse benchmarks demonstrate its superiority over existing methods. Code is available at https://github.com/LAMDA-CL/EMNLP2026-CRAM.

View source

Similar papers

#artificial intelligence Preprint Aug 2026

CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning

CoRe-MoE is proposed, a Compact Reusable MoE framework for parameter-efficient continual multimodal instruction tuning that improves final average performance over the strongest competing baseline by up to 5.90 points, while using less than 1% of the trainable parameters required by sequential LoRA for later tasks.

Run-Ze Liu, Nai-Bin Gu, Ming-Xu Ai et al. · 0 citations
#machine learning Preprint Oct 2026

Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

Learning new tasks in multimodal continual instruction tuning can cause catastrophic forgetting of previously acquired knowledge. Although multi-expert LoRA distributes learning across experts, updates to a dense low-rank expert can still affect many output features. We introduce Monarch-MoE, which replaces the dense f...

Xi Zhang, Bo-Yu Diao, Ze-Sen Chen et al. · 0 citations
#machine learning Preprint Sep 2026

ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning

Multimodal continual instruction tuning (MCIT) aims to enable multimodal large language models to acquire new capabilities from sequential tasks while preserving previously learned knowledge. Existing methods primarily mitigate catastrophic forgetting by constraining parameter updates or separating task-specific adapta...

Tao Hu, Zhi-Nuo Zhou, Xia-Liang Tong et al. · 0 citations
Conference Open access Sep 2026

Representation-Aware Modularity: Efficient Cross-Task Generalization for LLMs

RaMod, a Representation-Aware Modularity framework, is proposed, a Representation-Aware Modularity framework to extend the ReFT paradigm to CTG through two novel components: Dual-Modular Representation & Parameter Fine-tuning, which manipulates only a strategically chosen subset of hidden representations with modular i...

Zheng Gong, Ying Sun, Chao Wang et al. · 0 citations
#machine learning Preprint Sep 2026

PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation

PMOPD ( project-based Multi-Teacher On-Policy Distillation), which constructs subspace memories from the cumulative parameter displacements of different tasks and projects both gradients and optimizer updates to remove components that interfere with protected task directions, establishes geometry-aware optimization as...

You-Zhi Liu, Ruo-Bing Zheng, Bo-Yuan Tong et al. · 0 citations
Sep 2026

Interleaved Prompt Generation for Continual Instruction Tuning of Multimodal Large Language Model

Multimodal Large Language Models (MLLMs), pre-trained on vast image-text datasets, excel in zero-shot reasoning but face catastrophic forgetting when adapting to sequential tasks in Continual Instruction Tuning (CIT). While lightweight prompt-based methods offer a parameter-efficient strategy for sequential adaptation,...

Hong-Sheng Zhang, Zhong Ji, Di Wang et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.