Large-scale video diffusion models (V-DMs) have achieved remarkable text-to-video generation quality, yet their massive computational complexity makes deployment costly. Post-Training Quantization (PTQ) offers an appealing route to accelerate inference without retraining, but existing diffusion PTQ methods remain fragile on modern V-DMs. A key reason is that contemporary V-DMs are intrinsically multi-resolution due to multi-stage training, while most prior PTQ pipelines calibrate at a fixed resolution, causing suboptimal calibration signals and biased distributions under resolution changes. To address this gap, we propose S $^{2}$ Q-VDiT $^+$, a multi-resolution co-design PTQ framework from data, supervision, and quantizer perspectives. First, Denoising-Prior Based Multi-Resolution Sampling constructs resolution-consistent noisy latents by mapping to the clean space and re-noising, together with a trajectory-aware resolution policy across timesteps. Second, Structure-Aware Multi-Resolution Distillation enhances structural alignment via window-wise distillation and transfers resolution-aware spatial dependencies via multi-scale attention distillation. Third, Debiased Modulated Quantization mitigates skewed distributions using asymmetric weight quantization and a fuseable activation debiasing scheme. Extensive experiments on multiple state-of-the-art video generation models demonstrate that S$^{2}$ Q-VDiT$^+$ consistently outperforms strong PTQ baselines under W4A6 and W4A4, delivers up to $2.08\times$ end-to-end speedup, and reduces model storage and inference memory by up to $3.8\times$ and $2.1\times$, respectively.
Weilun Feng, Chuanguang Yang, Haotong Qin et al.· IEEE Transactions on Pattern...· 2 citations
Vision foundation models are increasingly reused as frozen backbones for downstream visual recognition, making parameter-efficient adaptation a central problem. Prompt-based adaptation, including Visual Prompt Tuning (VPT), provides a lightweight way to specialize these models, but its layer-wise behavior remains poorly understood: performance is sensitive to prompt depth, placement, and task distribution, and gains on standard in-domain benchmarks do not always translate into robust generalization. We argue that this limitation is not solely an optimization issue, but a layer-wise information allocation issue: existing prompt-based methods lack principled control over what prompt-conditioned representations should preserve, suppress, and propagate across depth. Inspired by the Information Bottleneck principle, we introduce Prompted Information Bottlenecks (PIB), a framework that regularizes layer-wise compression-sufficiency trade-offs and promotes a more coherent cross-layer information path. The key idea is that effective adaptation should be minimal yet sufficient, retaining task-relevant local evidence in earlier layers while progressively discarding nuisance factors and redundant details in deeper layers. Extensive experiments show that PIB achieves strong performance across 34 datasets, reaching 92.1% on FGVC, 93.01% on HTA, and 77.33% on VTAB-1k, while tuning only 0.35% parameters on average across the main settings. Beyond benchmark accuracy, PIB helps explain the non-monotonic behavior of prompt capacity scaling, reduces shortcut reliance, and improves robustness under distribution shift and fine-grained recognition settings. These results position PIB as both a practical method and an information-allocation perspective for adapting frozen vision foundation models. Our code is available at https://github.com/itsnotacie/MM-26-PIB
Yuqi Li, Xi Xiao, Yunbei Zhang et al.· arXiv.org· 4 citations
Distribution drift can substantially degrade the performance of deployed machine learning models; for example, accuracy on SST-2 can fall from 88% to 58%. Detecting such degradation is fundamentally challenging because deployment provides inputs but not labels, so the detection itself must be reference-free. We introduce DecayBench, the first reference-free, calibrated benchmark for evaluating drift detectors. DecayBench measures detector trustworthiness along five axes (calibrated, valid, timely, no-regret, adaptive), and compares ten existing detectors across ten NLP, vision, and multimodal datasets using paired-bootstrap significance testing. Evaluation on DecayBench shows that no existing detector is uniformly optimal. Motivated by this observation, we propose Alert, a label-free aggregation rule for drift detection. Unlike all competing combiners, it uses a label-free self-configuring selection rule with a no-regret guarantee. Alert has three contributions: (i) a dilution analysis yielding a self-configuring detector selection rule; (ii) a finite-sample conformal guarantee that controls the false-alarm probability on clean data at any prescribed level (e.g., 5%) for arbitrary score distributions; and (iii) a no-regret result: when no single detector dominates (constituents of comparable effect size, a condition checkable offline), Alert matches or beats the best constituent, being never significantly worse and sometimes better by a large margin; this holds across NLP, NLI, and vision (ResNet), with the largest gains under multimodal drift, and the proof identifies a dominant single detector (MMD on CLIP) as the only dilution exception. We prove the no-regret property and, across the benchmark, report its empirical counterpart, non-dominance under a paired bootstrap (Alert is never significantly worse than the best constituent), which at some operating points is statistically inconclusive rather than a strict win. Because Alert combines only embedding- and logit-based detector scores, it directly transfers across NLP, vision, and multimodal models. Empirically, Alert strictly improves over single-modality monitoring, increasing AUC by up to 25 points under mixed-modality drift and by approximately 50 points under cross-modal mismatch, where individual modality-specific detectors perform near chance. Alert also matches or outperforms the Fisher, Simes, Bonferroni, and median combiners, performs best under low-severity drift, and matches or surpasses early fusion (Concat-MMD) in both multimodal settings.
