A robust non-parametric Bayesian method known as the Bayesian adaptive tensor ring decomposition (BATR) method, which accomplishes an adaptive low tensor ring (TR) rank model by incorporating a more advanced generalized hyperbolic (GH) prior into the probabilistic framework, thereby facilitating automatic TR rank determination.
The advent of Large Language Models (LLMs) represents a leap toward leveraging AI for the benefit of humanity. However, the realization of this potential requires addressing the growing need for personalization and data privacy across diverse entities. Federated Learning (FL) offers a vital paradigm for developing private and personalized LLMs, yet its effectiveness is often limited by the inequality of computational resources at the edge. Existing federated Low-Rank Adaptation (LoRA) struggles to accommodate such heterogeneous hardware capacities, leading to information loss during model aggregation. To bridge this gap, this paper presents an adaptive aggregation framework designed to optimize federated LLM fine-tuning under these constrained conditions. The framework introduces a dynamic budgeting mechanism that quantifies personalization intensity through Frobenius norm divergence and allocates rank capacity accordingly. To resolve rank heterogeneity, a dimension-aligned strategy based on Singular Value Decomposition (SVD) is applied, enabling the consistent fusion of updates across diverse devices. Extensive experiments on the General Language Understanding Evaluation (GLUE) benchmark using Llama-3-8B achieve an average score of 0.6551 and a mean task ranking of 2.22. The results indicate that our method improves global generalization while preserving client-specific personalization in resource-limited environments.
Jinhua Chen, Yuning Qiu, Franck Junior Aboya Messou et al.· 2026 ITU Kaleidoscope - AI a...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
