Experiments show that SubZero+, an improved SubZero framework that improves stability in three complementary ways, consistently outperforms prior ZO baselines, enlarges the stable learning-rate range, and narrows the gap to first-order methods with minimal extra memory overhead.
Abstract
Zeroth-order (ZO) optimization enables backpropagation-free fine-tuning of large language models, but existing ZO methods suffer from high-variance gradient estimators, making convergence unstable and highly sensitive to learning rates. We propose SubZero+, an improved SubZero framework that improves stability in three complementary ways: (i) multi-query gradient estimation within layer-specific low-rank subspaces to reduce variance without exhibiting the multi-query paradox; (ii) a subspace Adam optimizer that performs adaptive updates using in-subspace multi-query gradient statistics; and (iii) a sign correction for QR-based subspace construction to ensure Haar-distributed projection matrices, eliminating implementation-dependent orientation ambiguity. Experiments on models from 1.3B to 32B across SuperGLUE, under both full-parameter tuning and LoRA, show that SubZero+ consistently outperforms prior ZO baselines, enlarges the stable learning-rate range, and narrows the gap to first-order methods with minimal extra memory overhead.
SCHUROPT is introduced, which analytically eliminates the suffix's optimal continuous response, yielding an exact groupwise quadratic with Schur-complement curvature, and achieves the highest mean zero-shot accuracy among the evaluated backpropagation free PTQ baselines.
Gunjun Lee, Sehwan Son, Younjoo Lee et al.· 0 citations
Post-training quantization (PTQ) is essential for deploying large language models (LLMs) under strict resource constraints. State-of-the-art PTQ methods quantize each layer with a single closed-form second-order solver: to remain analytically tractable, they heavily approximate the global loss (dropping cross-channel coupling, pooling output rows into groups), and they then freeze the resulting Hessian across the entire layer, with no way to refresh it as the loss landscape shifts column by column--a phenomenon we call information misalignment. We propose REAL-Q (Real-time E2E-loss Aligned LLM Quantization), a novel PTQ paradigm that breaks this compromise: instead of diluting the objective for the sake of analytic tractability, REAL-Q targets an end-to-end-aligned surrogate of the global loss and refines it via fine-grained, dynamic Block-wise Gradient Descent applied after every column block (128 columns). By coupling this fine-grained correction with a sliding window mechanism for smooth cross-layer transitions, REAL-Q effectively mitigates error propagation across the network. On LLaMA-3.1 (8B and 70B) and Qwen3 (0.6B-32B) at W4A16, REAL-Q reduces end-to-end KL divergence by up to ~49% relative to state-of-the-art globally-guided methods.
Qian Zhang, Yao-Ming Li, Zheng Tan et al.· 0 citations
Parameter-efficient fine-tuning (PEFT) methods adapt foundation models to specific domains by selectively updating only a small subset of critical parameters, significantly reducing computational costs. Among these methods, Sparse Low-Rank Adaptation (SoRA) has emerged as an effective approach for fine-tuning large language models. This paper investigates SoRA algorithm's performance in large-scale models, particularly analyzing learning dynamics and scaling effects in wide neural networks. We find that employing a uniform learning rate across all parameter matrices limits learning efficiency. To address this, we derive an improved learning rate configuration and propose SoRA+, an enhanced variant that introduces a hyperparameter to adjust learning rates proportionally. This modification aligns with theoretical optima and mitigates the suboptimality of the original SoRA method. Experiments show that SoRA+ improves fine-tuning performance by up to 2% while accelerating convergence.
Fine-tuning vision-language models such as CLIP typically requires backpropagation (BP) through the full model, which is infeasible when only forward-pass access is available, as is common for memory-constrained edge devices and proprietary model deployments. Prior BP-free, zeroth-order prompt-tuning methods avoid this requirement but often tune prompts in a single modality or optimize over a search space large enough that convergence requires thousands of forward passes, which is impractical under realistic query budgets. We propose ZOMP (Zeroth-Order Multimodal Prompt tuning), a query-efficient, fully forward-only method that tunes deep prompts in both the vision and text branches of a frozen CLIP model using simultaneous perturbation stochastic approximation. ZOMP combines three ingredients: a cross-modal low-rank reparameterization that ties the two branches through a shared factor and keeps the effective search dimensionality small, a gradient-correction momentum term that stabilizes the noisy zeroth-order estimate, and a budget-indexed rank schedule that unlocks capacity as the query budget is spent. Across 13 vision-language benchmarks under a matched 5,000-query budget, ZOMP consistently outperforms prior BP-free prompt-tuning methods in both few-shot accuracy and query efficiency, and it generalizes better across base-to-new, cross-dataset transfer, and out-of-distribution settings. Our results show that jointly exploiting multimodality and low-rank structure is an effective route to practical, query-efficient BP-free prompt tuning.
Sajjad Ghiasvand, Yifan Yang, Mahnoosh Alizadeh et al.· 1 citation
Parameter-efficient fine-tuning (PEFT) is a key technique for adapting large pre-trained language models to downstream tasks with minimal parameter updates. However, existing PEFT methods often suffer from slow convergence, gradient noise, and weak alignment between learned features and task semantics. To this end, we develop a low-rank adaptation framework based on gradient spectral alignment. Specifically, we (1) perform spectral decomposition of the pre-trained gradient covariance matrix and apply eigenvalue scaling to retain 95% of the spectral energy, optimizing parameter initialization; (2) design a dual gradient projector that combines orthogonal and spherical projections to suppress gradient noise, reducing its variance to 38.2% of standard LoRA; and (3) introduce a dynamic eigenvalue scaling mechanism that adaptively recalibrates principal component weights via a nonlinear scaling function, enhancing representational capacity. Experiments on the GLUE benchmark show that our method achieves performance comparable to full fine-tuning while training only 0.1% of parameters, outperforming AdaLoRA and DoRA by 6.2% and 5.3%, respectively. Additionally, our approach improves the mathematical reasoning accuracy of LLaMA-7B on GSM8K by 7.8%. This work provides novel insights into improving the efficiency, stability, and adaptability of PEFT frameworks. The code is available at https://github.com/rainylover/GSLoRA.
Test-time adaptation (TTA) aims to enhance the cross-domain performance of pre-trained models by adapting to unlabeled test data. While most existing TTA methods rely on backpropagation (BP) for finetuning, BP-free methods such as zeroth-order (ZO) methods are more desired in practical on-device scenarios. ZO methods rely only on forward computation, which can largely reduce the complexity and memory overhead of on-device deployment. However, ZO methods suffer from much higher variance compared with first-order methods in estimating the gradient. To address this, we propose an improved ZO method to substantially boost the performance of ZO optimization based TTA. First, we provide an observation to reveal the persistent low-rank Hessian structure of the loss during the adaptation process. Based on this insight, we then propose a loss-landscape curvature-aware zeroth-order (CAZO) method, which leverages a sliding-average estimation of the diagonal Hessian to construct a covariance matrix for anisotropic perturbation sampling. CAZO operates by freezing pretrained weights and optimizing minimal adapter parameters via forward-only passes based gradient estimation, which can substantially reduce the memory overhead compared to BP-based methods. Extensive experiments demonstrate that CAZO significantly outperforms existing TTA methods, achieving state-of-the-art performance while maintaining an excellent balance between accuracy and memory efficiency. Code is available at https://github.com/Hollyming/CAZO.
Junming Zhang, Shuyu Yin, Peilin Liu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
