Skip to content
Preprint

Token Communication for Multimodal Large Language Model

Aug 2026 · 0 citations · 42 references
Engineering

TL;DR

This paper investigates how to reduce the amount of transmitted data during interactions with MLLMs while preserving their multimodal understanding performance, and proposes a token communication framework tailored to MLLMs.

Abstract

With the broad success of the Transformer architecture, token is becoming a new basic information processing unit. This trend is especially evident in multimodal large language models (MLLMs), where both visual and textual information are represented and processed as tokens. With the rapid deployment of MLLMs, the efficient transmission of tokens has become increasingly important. This paper investigates how to reduce the amount of transmitted data during interactions with MLLMs while preserving their multimodal understanding performance. To address this problem, we propose a token communication framework tailored to MLLMs. In the proposed framework, a neural codec is integrated into the vision tokenizer to control the number of transmitted bits. At the receiver, the decoded latents are processed through two paths. The decoder reconstructs image as a reconstruction prior, while the adapter converts latents into visual tokens and injects them into an intermediate layer of the vision tokenizer. To make the injected tokens suitable for MLLMs, we further design a two-stage visual-language semantic alignment training scheme. The adapter is first warmed up by a distillation loss and then aligned with textual semantics through an alignment loss. An adaptive adapter is also introduced through feature-wise linear modulation, allowing one adapter to support multiple codec rates. Extensive simulations on various MLLM benchmarks show that, under the same amount of transmitted data, the proposed scheme achieves better task performance than other image processing schemes for MLLMs.

View source

Similar papers

Mizar: A 159M-Parameter Audio-Language Model for Audio Understanding

Mizar, a 159.3M-parameter ALM, is introduced, a recipe that brings together architecture, data, and three-stage training to build Mizar, a 159.3M-parameter ALM that surpasses the previous best-performing ALM below 200M parameters on all three benchmarks.

Kai-Yang Li, Shaobo Han, Yue Tian et al. · 0 citations
#artificial intelligence Preprint Sep 2026

VoT: Vision-of-Thought for Unified Multimodal Representation Alignment

Vision-of-Thought (VoT), a framework that introduces a discrete visual-thinking layer between vision-language models (VLMs) and diffusion transformers (DiTs) that improves semantic alignment and provides a structured interface for interpretable and controllable generation is proposed.

Jingxiang Sun, Chao Liao, Zheng-Xiong Luo et al. · 0 citations

Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models

This work proposes $\delta$-Vision, which replaces repeated Transformer evolution of visual tokens with lightweight low-rank adapters that construct layer-wise visual memories while preserving all visual tokens for text retrieval, and achieves higher accuracy than visual token pruning baselines at comparable or lower c...

Jing-Di Lei, Junxian Li, Di Zhang et al. · 0 citations
Preprint Aug 2026

Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models

The Visual Information-Guided Sampler (VIG-Sampler), which prioritizes tokens based on their attention to image tokens, and imposes a constraint that penalizes candidate tokens whose image-attention distributions are similar to those of previously selected tokens, thereby increasing the information gain of the decoded...

Insu Lee, Woo-Soon Park, Wonseok Shin et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.