Jul 2026· IEEE Transactions on Neural Networks and Learning Systems· Vol PP, pp. 1-15· 0 citations
Medicine
TL;DR
Experiments show that ChACo improves over prior local-learning baselines, transfers across WAN and ResNet convolutional architectures, and narrows the gap to matched backpropagation (BP) references, while end-to-end BP remains stronger in the most depth-dependent settings.
Abstract
Local layer-wise learning offers modular optimization, layer-level transparency, and training without end-to-end error transport. However, its scalability remains limited by three coupled difficulties: local objectives can be weak or poorly aligned with the final task, shallow layers are often forced into premature fine-class discrimination, and fully local optimization is sensitive to activation and variance drift across depth. In this work, we present ChACo, a channel-wise adaptive competitive framework for fully local learning in convolutional networks. Each layer is trained with a local discriminative objective, while the internal block-to-class association is allowed to vary with depth. Early layers can map fewer competitive blocks to fine-label logits through a learned association policy, whereas later layers can recover direct fine-class competition as a special case. This design reduces the shallow-layer channel burden in many-class settings while preserving a direct local path to the final label space. The framework is supported by an analysis of local optimization dynamics, showing why even-power goodness functions and the ordering of rectification, convolution, and normalization are important for stable activation conditioning. We further incorporate lightweight stabilization components, including block-wise normalization and modular training schedules. Experiments on standard and many-class image-classification benchmarks show that ChACo improves over prior local-learning baselines, transfers across WAN and ResNet convolutional architectures, and narrows the gap to matched backpropagation (BP) references, while end-to-end BP remains stronger in the most depth-dependent settings. The source code of ChACo is available at https://github.com/andreaspapac/ChACo.
Learning rate (LR) initialization and decay remain important factors in the optimization of deep vision networks. Although these models exhibit a clear hierarchical structure, training typically starts from a single global learning rate, with little explicit consideration of stage depth. This paper investigates a simple depth-aware alternative. We propose Asynchronous Learning Rate (ALR), which assigns depth-dependent initial learning rates to network modules according to their topological depth, and Smoothed Synchronous Decay (SSD), which coordinates the subsequent decay of heterogeneous parameter groups. A linear depth rule is adopted as a low-complexity, monotonic parameterization that is straightforward to implement across hierarchical architectures. The proposed strategy is evaluated on image classification and semantic segmentation benchmarks, including CIFAR-100, Mini-ImageNet, ImageNet-1K, Pascal VOC 2012, and LiTS. Additional ResNet-50 evaluations under benchmark-specific training protocols show that depth-scaled ALR improves Top-1 accuracy from 77.618% to 78.736% on native CIFAR-100 across five seeds and from 76.33% to 78.58% on ImageNet-1K. The results indicate that ALR generally outperforms uniform learning rate initialization, whereas SSD is most effective when combined with depth-aware initialization, particularly when layer-wise learning rates would otherwise follow misaligned decay trajectories. These findings suggest that a simple hierarchy-aware learning rate design can serve as an effective optimization refinement for hierarchical vision models.
Qiang He, Qiu Zong, Yi-Qi Wang et al.· IEEE Access· 0 citations
Continuous advancements of deep learning techniques have profoundly influenced Artificial Intelligence (AI) for visual classification through shifting the field from manual feature engineering to autonomous, hierarchical feature learning. On the contrary, the traditional mechanisms for visual classification relied on various challenges, including large data requirements, computational demands, model interpretability issues, and bias concerns, which severely limited accurate classification. Therefore, the research proposes the Fight-Or-Free Optimized Distributed Patch-Wise Attention-Driven Bidirectional Long Short-Term Memory Network (F2DPAB-Net) for visual classification. The Fight-Or-Free Optimization (F2Opt) algorithm significantly tunes the hyperparameters using stochastic behaviors, potentially improving convergence speed and providing a balance between local exploitation as well as global exploration. Integration of patch-wise triplet attention fusion mechanism offers parallel processing, making the model more efficient in learning long-range dependencies that substantially increase the significance while training. On top of that, utilization of multimodality features enables the model to process and understand different modalities that achieve a more comprehensive interpretation of information and improve generalization. Overall, the proposed F2DPAB-Net outperforms existing methods, thus attaining a maximum of 0.981 Cohen’s Kappa Score, 0.96 MCC, and 0.984 NPV using the COCO dataset, respectively.
M. Kiran Kumar Goud, B. Nandini· International Journal of Ima...· 0 citations
It is shown that a principled design space for LoRA initialization and curvature preconditioning should be treated as a tunable dimension rather than a fixed design decision, and a deployable, search-free variant, ULoRA-Auto, selects per-layer exponents from measured spectral statistics, approaches this upper bound at no additional search cost, and ranks at or near the top among deployable LoRA methods.
Low-Rank Adaptation (LoRA) is a prominent fine-tuning method for large models, achieving competitive performance with reduced memory overhead. However, a persistent performance gap remains between LoRA and full fine-tuning. Recent studies have sought to narrow this gap by employing one-step gradient approximations of pretrained weights to align LoRA updates with the principal directions or intrinsic dimensionalities of full fine-tuning updates. Nevertheless, these approaches fail to capture the full dynamics of the gradients. In this paper, we propose LoRA-GA$^2$, an effective fine-tuning algorithm that fully leverages multi-step gradient information. Specifically, we introduce a lightweight probe for multi-step gradients of pretrained weights that incurs no additional GPU memory cost and only marginal time overhead. We further employ a spectrum-aware, importance-based rank allocation and optimal initialization derived from multi-step gradients. Extensive experimental results demonstrate that LoRA-GA$^2$ consistently outperforms existing LoRA variants while preserving the efficiency advantages of vanilla LoRA. For instance, LoRA-GA$^2$ surpasses the leading baseline by an average of 0.66 points on the GLUE benchmark, and outperforms the strongest baseline by 1.03 points on GSM8K and 0.87 points on HumanEval, respectively.
Experimental findings show that pretrained models outperform those trained from scratch in terms of accuracy, convergence speed, and robustness, and a unified framework is proposed to integrate both processes in a deep learning pipeline.
Kwame Nkosi· International Journal of App...· 0 citations
Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR convention, applying a single global learning rate across every rank-one component of every adapter. We show that this convention overlooks substantial within-module heterogeneity, where the rank-one components of a LoRA adapter update at highly uneven rates and low-velocity modules converge to concentrated singular spectra that underutilize the nominal rank budget. To address this, we propose an adaptive anisotropic learning-rate model that assigns each rank-one component its own effective learning rate, computed online from training-time signals and mean-normalized per module to preserve the global LR budget. AnLR-LoRA instantiates this model with two signals available during AdamW optimization, namely function-space velocity and Adam SNR, as a lightweight scheme with no extra trainable parameters. Across commonsense reasoning, natural language generation and visual instruction-tuning benchmarks, AnLR-LoRA consistently improves over LoRA while encouraging broader use of rank capacity, with gains that remain robust across a wide range of global learning rates and transfer cleanly to other LoRA variants.
Huiyi Wang, Daijiao Liu, Lina Yao et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
