Replay-free Sequential Fine-tuning of Medical VLMs

Hefei Li Yuhui Zhang Xiaohan Wang S. Yeung-Levy Josh Achiam Steven Adler Lama Sandhini Agarwal Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt S. Altman Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr Yana Hasson Arthur Lenc Katherine Mensch Malcolm Millican Huanran Chen Yinpeng Dong Zeming Wei Yao Yichi Huang Han Zhang Su Jun Zhu Adrito Das D. Khan Dimitrios Psychogyios Haotian Liu Chunyuan Li Qingyang Wu

· 0 citations · 15 references

TL;DR

The streamlined, replay-free approach proves highly effective on a new continual learning benchmark and a practical path toward building prehensive, continually-learning medical VLMs and advancing the development of medical AI.

Semantic Scholar

View source

Similar papers

Open access Jul 2026

Biological Continued Pretraining Reshapes the Capability Profile of a Foundation Model Without Catastrophic Forgetting

It is widely assumed that continued pretraining (CPT) on a narrow, out-of-distribution corpus such as raw biological sequence must trade away a general-purpose model’s broad competence — the “alignment tax” or catastrophic-forgetting intuition. We test this directly, without any new training, by re-analyzing three chec...

Liang Wang · bioRxiv · 0 citations

Jul 2026

An Empirical Analysis of Continual Learning for Heterogeneous Medical Visual Question Answering

A systematic evaluation of CL for MedVQA across diverse clinical objectives, including classification, multi-label classification, detection, cell counting, and report generation suggests that existing CL methods struggle to maintain stability-plasticity balance when tasks with different objectives and supervision form...

Mai A. Shaaban, Tausifa Jan Saleem, Alaa Mohamed et al. · arXiv.org · 0 citations

Jul 2026

The Art of Not Forgetting A Local Learning Architecture for Continual Learning

The results suggest that the combination of sparse representations, local learning, and persistent memory is a promising direction for continual learning, while motivating further investigation into the respective roles of learning rules, representations, and architectural design in mitigating catastrophic forgetting.

Ashmith Atmuri, Yashaswini Rao Bhogarajula · arXiv.org · 0 citations

Jul 2026

TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models

TCLA corrects inference logits based on a small set of support samples, boosting pretrained VLMs performance by improving inter-class deconfusion and reducing domain shift, and consistently improves OOD performance of Medical VLMs and outperforms existing training-based adaptation methods.

Tianyou Jiang, Ziyu Zhou · arXiv.org · 0 citations

Jul 2026

The Art of Not Forgetting

We introduce CMP (Cognitive Memory Primitive), an architecture that represents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns entirely through local, gradient-free updates, with no backpropagation anywhere in the network. We use this architecture to test a specific hypothesi...

Ashmith Atmuri, Akshay Kumar, Yashaswini Rao Bhogarajula · arXiv.org · 0 citations

#small language model Preprint Aug 2026

Unifying Detection and Adaptation in Task-Free Continual Learning

Experiments show that FiUni can effectively infer latent batch-level task affiliations and achieve competitive performance against advanced task-aware CL methods with fewer trainable parameters.

Dezheng Han, Anlan Zhang, Zhiwu Zhu et al. · 0 citations