BPG: Balancing Plasticity and Generalization for Domain Incremental Learning

Single Domain Generalization Benchmark: Current State and Challenges

Recent advances in deep learning have enabled remarkable success across computer vision tasks using massive datasets, yet the persistent challenge of domain shift-where training and testing distributions differ-continues to degrade model performance. While Domain Generalization techniques aim to learn invariant representations across multiple domains, their reliance on diverse training domains limits applicability in settings where multiple source domains are not available. This survey examines Single Domain Generalization (SDG) approaches that leverage targeted inductive biases to enhance out-of-domain (OOD) performance using a single source domain, addressing key questions regarding their effectiveness across various domain shifts, different model architectures such as CNNs and Vision transformers, and the impact of shape bias for OOD generalization. These questions are crucial as they directly inform the development of robust models that can reliably operate under real-world conditions, guiding the research community in overcoming limitations of current techniques. Our findings indicate that although SDG methods improve performance under highly domain distinguishable stylized shifts, they exhibit limited robustness to background, correlation and corruption shifts on larger datasets, and increased shape bias does not consistently yield enhanced OOD performance, thereby underscoring the need for further research into developing more resilient and generalizable models. For reproducibility, the implementation and code used in our benchmark are available on our official https://github.com/Kavindya98/SDG-Benchmark GitHub repository.

Representation-Steered Incremental Adapter-Tuning for Class-Incremental Learning with Pre-Trained Models

Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms

Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate domain experts and subsequently consolidating them. We organize three fusion paradigms by the artefacts they reuse: Merge combines expert task vectors, Mix RL pools their datasets, and multi-teacher on-policy distillation (MOPD) uses both. Because they have largely been studied in isolation, how they compare and how to choose among them remain unclear. We compare all three using shared experts and data across model scales and a multi-domain benchmark suite. Although their average performance differs by at most 1.4 points, the gap reaches 8.6 points on a single benchmark, with domain-level variation tracking cross-domain relations visible in task-vector geometry. Training dynamics expose distinct constraints: Mix RL depends on domain mixture proportions, MOPD remains bounded by its teachers, and Merge compresses all expert updates into one. All three improve single-sample accuracy without measurable gains in solution coverage or losses in held-out capabilities. These results yield a practical guideline: use Merge when experts already exist and cheap fusion is paramount; Mix RL when training a unified model without experts, with domain proportions adjusted for cross-domain transfer; and MOPD when preserving domain-specific gains matters more than surpassing teachers or minimizing end-to-end cost.

Miles: Metric Learning With Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning

Class Incremental Learning (CIL) aims to learn new concepts consistently from a data stream without forgetting. Unlike typical CIL methods which need to learn a model from scratch, pre-trained model (PTM) can easily adapt to a new task with fine-tuning. However, existing PTM-based CIL methods fail to achieve a trade-off between performance and computational expenditure, i.e., they either adopt the same parameter space so that leading catastrophic forgetting, or expand a new branch for each task but adding more computational cost. To this end, we propose MetrIc Learning with Expandable Subspace (Miles) to harness the prior information within pre-trained knowledge, thereby orchestrating an efficient expansion of the parameter space through guided optimization. Specifically, it decouples the learnable modules with the pre-trained model, exploiting prior information from intermediate features of the backbone network to enable more flexible parameter expansion. Then, a central loss is adopted to guide the new category to cluster towards the corresponding prototype in the new task subspace while incorporating an auxiliary distance regularization term to maintain metric equilibrium across tasks. Extensive experiments on six benchmark datasets demonstrate that Miles achieves state-of-the-art performance in various CIL settings.

Rethinking machine learning for engineering

A data maturity-deployment scale framework is introduced, arguing that engineering AI should be understood not as a search for a single best model, but as an evolving intelligence strategy that co-adapts with system maturity, operational context, and deployment scale.

Learning the Pareto Frontier of Predictive Models under Distribution Shift

Modern machine learning pipelines increasingly rely on reusing pretrained and foundation models across downstream tasks. These pretrained models can differ not only in performance but also in how they can be used: some only provide black-box predictions, while others may permit white-box access to internal representations that can be probed or fine-tuned. When deployed to the target domain in the presence of distribution shift, no single strategy, including zero-shot application, fine-tuning, or directly training a target-specific model, is uniformly the best. In this work, we propose Frontier Learning, a framework that treats a library of candidate models spanning different training histories and access regimes as complementary sources of information rather than mutually exclusive alternatives. Frontier Learning constructs a unified target-domain feature by concatenating internal representations from white-box candidates as well as prediction outputs from black-box candidates, then fits a lightweight, regularized supervised learner on this concatenated representation using labeled target data. Because the resulting hypothesis class contains predictors obtained by zero-shot reuse, fine-tuning, and direct training as special cases, empirical risk minimization over the frontier learner is guaranteed to be no worse, on the training sample, than any individual baseline. We evaluate the framework in simulations spanning varying degrees of source-target compatibility and in two real-world distribution-shift settings: visual domain adaptation on DomainNet/VisDA and clinical mortality prediction across intensive care unit domains using MIMIC-IV-Notes. Across all settings, Frontier Learning matches or outperforms the strongest individual reuse strategy, with the largest gains arising precisely when no single baseline is reliable across the range of shift considered.