Adapting pretrained models to downstream tasks with limited data has become a central paradigm in modern deep learning. Yet, despite its widespread practical success, how fine-tuning leverages information from pretraining remains poorly understood theoretically. We study fine-tuning from pretrained weights through the...
A.-E. Decleves, Etienne Boursier, Nicolas Flammarion· 0 citations
An information-theoretic framework for generalization in next-token prediction under temporally dependent data is developed, revealing the roles of context length, model complexity, sample size, margin, and temporal mixing and yielding informative guarantees for deterministic algorithms over continuous hypothesis space...
It is demonstrated that pre-training can induce an implicit bias with a clear statistical advantage over random initialization, enabling feature learning from scarce fine-tuning data.
Flow matching (FM) has recently emerged as a promising framework for generative modeling due to its conceptual simplicity and strong empirical performance. In FM, samples are transported along a vector field parameterized by a neural network, inducing a probability path that evolves from a simple noise distribution to...
Ze-Yuan Yu, Zhi Chang, Shi-Wei Lan· 0 citations
Reach audiences
Advertise in front of researchers, engineers, and readers.
Tail-Corrected Top-$k$ On-Policy Distillation (TT-OPD) is proposed, which preserves the advantages of TK-OPD, including rich distributional supervision and low computational cost, while providing an unbiased estimator of the gradient of the reverse KL divergence.
Lin-Jian Meng, Si-Yuan Gan, Yu-Hang Li et al.· 0 citations
An unsupervised framework for functional data that spans both univariate and multivariate cases, with emphasis on the multivariate setting, and factorizes the latent space into separate amplitude and phase embeddings derived from all channels is proposed.
Multivariate time series anomaly detection (MTAD) is crucial for ensuring the safe and reliable operation of complex systems. Many existing methods learn normal patterns by training reconstruction or forecasting models on predominantly normal data. However, a large portion of these approaches rely on deterministic mode...
W. Koo, Jaeyeong Lee, Taeseong Yoon et al.· 0 citations
What determines the unavoidable sample cost of learning cyclic causal structure? For cyclic linear non-Gaussian models, we study exact condensation recovery from observational data: identifying the strongly connected component (SCC) partition and all edges between components. We establish the first information-theoreti...
Non-negative matrix factorization (NMF) is a singular statistical model whose Bayesian asymptotics are governed by the real log canonical threshold (RLCT). We study the local geometry of the factorization map and derive an upper bound for the RLCT of NMF. Let $H$ be the model inner dimension and $H_0$ the non-negative...
Testing whether two populations of networks share the same edge probabilities is a basic problem in network inference. How hard it is depends on the norm used to measure the difference. For the inhomogeneous Erd\H{o}s--R\'enyi (IER) model, the optimal sample complexity is known for every integer $L_r$ norm and for $1\l...
Inverse Optimal Transport (OT) based methods for representation learning learn representations such that the global OT coupling between a pair of data marginals in the representation space, concentrates on the positive pairs. This is in contrast to previous methods that primarily focused on pairwise matching. However,...
Ngoc-Hai Nguyen, Thuan Nguyen, Prakash Ishwar et al.· 0 citations
Estimating causal effects with continuous treatments in observational studies is challenging due to confounding, model misspecification, and high-dimensional covariates. We propose the Weighted Spline-Expanded Network (WSENet), an end-to-end neural framework that addresses these challenges by combining covariate balanc...
Shu-Cheng Liu, Chan Park, Guan-Hua Chen· 0 citations
Jennifer Neville did not want to go into computer science—but that’s exactly where she landed. Neville discusses the starts and stops that led to her professional sweet spot and her work identifying “surprising failures” making it hard for AI to handle complexity. The post What AI gets wrong and what failure teaches us appeared first on Microsoft Research.
