Preprint

Local Margin Restoration for Test-Time Adaptation of Vision-Language Models

Yan Huang Guowei Wang Xu Wang Kangjun Liu Xin Lin

Aug 2026 · 0 citations

Computer Science

TL;DR

Local Margin Restoration (LMR) is proposed, a lightweight, one-step TTA framework that consistently outperforms state-of-the-art TTA baselines, proving exceptionally robust and efficient even in challenging low-batch test-time regimes.

Abstract

Vision-language models (VLMs) such as CLIP exhibit remarkable zero-shot capabilities, yet their performance frequently degrades sharply under unexpected test-time distribution shifts. While Test-Time Adaptation (TTA) offers a promising solution, continuously adapting VLMs over an unlabeled test stream presents fundamental challenges. Conventional top-1-centric updates often reinforce errors by corrupting the local semantic geometry among related classes, while iterative adaptation exacerbates progressive bias accumulation, ultimately driving the model toward mode collapse. To overcome these coupled vulnerabilities, we propose Local Margin Restoration (LMR), a lightweight, one-step TTA framework. At the sample level, our Protected Margin Restoration (PMR) objective recovers local semantic geometry by shielding plausible near-top candidates from external hard negatives. Concurrently, to combat stream-level degradation, we introduce a dual-stage stabilization mechanism, featuring an Adaptive Margin (AM) controller and Bias Correction (BC), to dynamically disrupt progressive bias accumulation and prevent mode collapse. Extensive experiments on CIFAR-C, ImageNet-C, and ImageNet variants demonstrate that LMR consistently outperforms state-of-the-art TTA baselines, proving exceptionally robust and efficient even in challenging low-batch test-time regimes. Our code is available at https://github.com/DennisHuangYan/LMR.

arXiv Semantic Scholar

View source

Similar papers

Preprint Jul 2026

Robustifying Vision-Language Models via Test-Time Prompt Adaptation

This work proposes RITA, a Robust test-tIme prompt-TAdaptation framework that shifts from sample-level estimates to distribution-level alignment, and employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal seman...

Xingyu Zhu, Huanshen Wu, Shuo Wang et al. · 1 citation

Review Jul 2026

Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions

This comprehensive survey formally defines the CTTA problem, analyzes the diverse continual domain shift patterns that characterize different evaluation protocols, and proposes a hierarchical taxonomy that categorizes existing methods into three families: optimization-based strategies (entropy minimization, pseudo-labe...

Sarthak Kumar Maharana, Shambhavi Mishra, Yunbei Zhang et al. · 2 citations

#artificial intelligence Preprint Aug 2026

From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model

This work proposes a principled VLM TTA method called \algname, and theoretically reveals that the InfoNCE loss can be neatly reformulated as a Wasserstein OT formulation, thereby unifying the objectives of the inference and adaptation of VLMs to achieve their mutual benefits.

Qi Yu, Zhichen Zeng, Katherine Tieu et al. · 0 citations

Preprint Aug 2026

Towards Purified Multi-Label Test-Time Adaptation of Vision-Language Models

PuRF is introduced, a novel PuRiFication-driven cache-based method for multi-label test-time adaptation of vision-language models that consistently outperforms state-of-the-art methods on ViT-B/32 across five datasets.

Yiwen Liang, Hui Chen, Yizhe Xiong et al. · 0 citations

Preprint Jul 2026

TestMate: Test-Time Domain Adaptation Aided by Lightweight Vision Foundation Model

TestMate is introduced, a novel, real-time, and backpropagation-free TTDA framework that enables immediate adaptation from the first frame, inherently avoids catastrophic forgetting and effectively preserves fine object details and boundaries, even for small objects.

Dimitrios Fotiou, V. Mygdalis, I. Pitas · 0 citations

Degradation-Consistent Test-Time Adaptation for All-in-One Image Restoration

Extensive experiments across multiple tasks and challenging domain shifts demonstrate that DCTTA consistently outperforms state-of-the-art AiOIR baselines, achieving up to +4.57 dB PSNR improvement on the Rain100H dataset.

Ni Tang, Shenghao Nie, Xiaotong Luo et al. · 0 citations