Experimental results demonstrate that proposed Bayesian domain weighting method could achieve stable and efficient domain weights learning, and identifies optimal mixtures while consuming substantially less data than search-based function-fitting methods, revitalizing optimization-based domain weighting for large-scale applications.
Abstract
The performance of Large Language Models (LLMs) is fundamentally influenced by the distributional composition of multi-domain pre-training data. While manual heuristics were prevalent in early models, they increasingly fail to capture the intricate synergies between domains as data complexity grows. To overcome the issue, a dominant approach seeks to fit a proxy function mapping between domain weights and their corresponding validation losses, and then find the optimal domain weights to minimize validation losses. These methods rely on strong structural assumptions, such as rank invariance or scaling laws, which are often violated, resulting in non-negligible estimation bias. A promising approach is to directly optimize the weighting scheme from data. However, it suffers from unstable optimization trajectory and prohibitive computational overhead, limiting its potential to search better domain weights configurations. This paper presents a Bayesian domain weighting method to infer the weights from a Dirichlet distribution via introducing Gamma prior information learned from observations. Experimental results demonstrate that proposed method could achieve stable and efficient domain weights learning, and identifies optimal mixtures while consuming substantially less data than search-based function-fitting methods, revitalizing optimization-based domain weighting for large-scale applications.
A range of methods aim to enhance the performance of vision-language models (VLMs) at test time. Among them, transduction has emerged as a promising paradigm due to its strong compatibility and efficiency. However, realistic evaluations often involve highly imbalanced class distributions, which cause performance degradation or even collapse. In this work, we systematically revisit transduction from the perspective of penalized likelihood estimation (PLE), showing that PLE with a KL-divergence anchor term naturally yields an adaptive shrinkage behavior between prior anchors and empirical estimates. From this viewpoint, the brittleness of transductive methods can be attributed to the absence of anchoring mechanism and static modeling of the shrinkage strength. Therefore, we propose Mixture of Von Mises-Fisher Models with Dynamic Shrinkage (MOON). MOON is built upon a mixture of von Mises-Fisher distributions to model feature representations on the unit hypersphere. To handle imbalance, MOON dynamically adjusts the shrinkage strength using zero-shot priors at both instance and class levels. Thus, it suppresses unreliable assignments and prevents harmful updates from outlier classes, thereby mitigating negative transfer. MOON is model-agnostic, training-free, and requires no task-specific hyperparameter tuning. Extensive experiments further validate the advantage of MOON in both performance and efficiency. Our code is available at https://github.com/walawalagoose/MOON
Jiazhe Huang, Zhiming Liu, Changhu Wang et al.· arXiv.org· 2 citations
Causal Structure-guided DRO (CS-DRO) is proposed, which estimates a directed acyclic graph (DAG) that encodes the predictive relationships between representations and labels, serving as a proxy for causal structure shared across source domains.
Seonggyeom Kim, Eunjung Choi, Dong-Kyu Chae· Proceedings of the 32nd ACM...· 0 citations
The concept of ranked sparsity, originally introduced in the context of penalized regression, arises in modeling applications when an expected disparity exists in the quality of information between different feature sets. Its presence can cause traditional and modern model selection methods to fail because such procedures commonly presume “covariate equipoise”—that each potential parameter is equally worthy of entering into the final model. However, this presumption does not always hold, especially in the presence of derived variables or with highly disparate feature sets (i.e., multi-modal data). For instance, when all possible interactions are considered as candidate predictors, the sheer number of them grossly inflates the number of false discoveries, resulting in unnecessarily complex and difficult-to-interpret models with many (truly spurious) interactions. In this work, we motivate a ranked sparsity extension to the Bayesian Information Criterion (RBIC) that requires a stronger level of evidence in order to allow certain variables (e.g., interactions vs main effects and genetic vs clinical covariates) into a model. We compare the performance of RBIC relative to competing methods for selecting polynomials and interactions in a simulation study and in two applications, showing that stepwise selection guided by RBIC produces better-predicting, more transparent models (with fewer false interactions) compared to existing alternatives.
Ryan A. Peterson, Sarah M. Bird, Logan M. Harris et al.· Entropy· 0 citations
Influence functions provide a principled framework for tracing model predictions back to training data, yet existing methods remain impractical for large language models due to prohibitive storage and I/O costs. Prior approaches face two critical barriers: the Fidelity Gap from coarse curvature approximations, and the I/O bottleneck from materializing per-example gradients, which renders million-scale datasets infeasible. We propose StructInf, a structure-aware influence estimation framework that addresses both gaps through system-algorithm co-design. For fidelity, we first replace the indefinite Hessian with an adaptive block-diagonal damped empirical Fisher surrogate, computed via quadratically convergent Newton--Schulz iterations. For efficiency, we then introduce a streaming algorithm that eliminates per-example gradient storage entirely, reducing space complexity from O(N • d) to O(d) and enabling processing of million-scale training instances on consumer hardware, where N denotes the number of training instances and d the parameter size. We further identify and resolve Split-View Bias, a distributed pitfall where naive averaging of local curvatures fails, via a distributed gradient calibration strategy that synchronizes global statistics with minimal communication. Finally, we conduct comprehensive experiments to evaluate StructInf against other baselines. Unlike prior methods requiring hours of pre-computation or TB-scale storage, StructInf enables real-time data selection during training. Across GLUE and instruction-tuning benchmarks, StructInf achieves up to +26% AUC on MRPC and average 7× speedup for 7B-scale models, making high-fidelity influence analysis practical on a single RTX 4090. These results suggest that practical IF for LoRA-tuned LLMs benefits from joint optimization of curvature modeling, memory management, and distributed coordination.
Mengyi Yan, Yaoshu Wang, Guangyi Zhang et al.· Proceedings of the 32nd ACM...· 0 citations
This work presents a framework for learning continuous latent representations of admissible partial differential equations by embedding a scientific inductive bias directly into the training distribution, and shows that embedding a scientific inductive bias in the training distribution enables the learning of compact and geometrically meaningful hypothesis manifolds.
James Crowley, Faez Ahmed, A. van Beek· 0 citations
Machine learning models continue to face challenges in out-of-distribution (OOD) generalization, where domain generalization (DG) aims to improve performance on unseen domains under distributional shifts. A prevalent paradigm in DG focuses on learning domain-invariant feature representations. However, feature representations from existing methods often exhibit weak interpretability. To bridge this gap, we propose Sparse Additive Domain Generalization (SpADG). We incorporate an additive structure into the DG framework and employ ℓq,1 -norm regularization to induce sparsity, thereby enabling structured feature selection and enhancing interpretability. We present two distinct realizations: an additive kernel-based formulation and a neural additive model-based approach. The former leverages the representer theorem for flexible data adaptation, while the latter learns nonlinear shape functions. Theoretically, we derive generalization error bounds for both realizations and prove the feature selection consistency of our method under rate-scaled regularization condition. Empirical evaluations on synthetic and real-world datasets validate the effectiveness of SpADG, particularly its robustness in high-dimensional settings.
Jiayi Wang, Han Li· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
