Optimal Rates for Learning with Monotone Adversaries

Bagging Robustly Learns VC Classes with Linear Sample Complexity

It is proved that VC classes are adversarially robustly learnable with sample complexity linear in the VC dimension $d$, providing an exponential improvement over the previous upper bound of Montasser, Hanneke, and Srebro (2019).

When Does More Correct Data Hurt? Insertion-Stability and the Limits of Dimension-Based Theory

Adding data known to be correct ought to be safe. Not always. Larsen, Pabbaraju and Shetty model the failure with a monotone adversary, which reads an i.i.d. training sample and may append as many further examples as it likes, provided the target hypothesis labels them all. Mehrotra has since settled the cost, showing that for classes of VC dimension d>= 2 no learner can guarantee expected error better than Theta((d/n)log(en/d)), a logarithmic factor above the clean PAC rate. Because that rate is a worst case over all classes, it says nothing about which classes actually suffer the penalty, and the answer turns on the learner. We call a learner insertion-stable if feeding it more correctly labeled examples can only shrink the region where it errs. Such learners are immune to the adversary, since on any given sample the risk after insertions never exceeds the risk on the clean part alone, however much is added and however cleverly it is chosen. High- probability guarantees carry over unchanged, and because Closure is insertion-stable every intersection-closed class keeps its clean rate of E[Err]<= (21d+34)/n. Immunity is not something the classical dimensions can predict. Two classes can agree on VCdim = Ldim = 2 and still split, one at Theta(1/n) and the other at Theta(log(en)/n), while intervals have unbounded Littlestone dimension and are immune anyway. On Mehrotra's hard class we prove more than the failure of a single algorithm, showing that no monotone permutation-invariant compression scheme of any finite size attains the clean rate. The question is therefore not whether a class is hard, nor whether a learner is good, but whether the two suit each other. Given an insertion-stable learner that is optimal on clean data, correct additions are free, and without one the cost belongs to the class, so changing the learner will not avoid it.

Optimistic Rates for Multiclass PAC Learning

Worst-case multiclass bounds do not become smaller when the best classifier is already nearly correct: what is missing is an optimistic rate, a guarantee whose fluctuation scales with the oracle risk itself. For a class of Natarajan dimension $d_N$ and Daniely-Shalev-Shwartz dimension $d_{DS}$, the optimal excess risk is known at the two endpoints ($d_{DS}/n$ realizable, $\sqrt{d_N/n}+d_{DS}/n$ agnostic [HMZ24, CEH+26, Pab26]) and open in between. We close the gap: at every fixed oracle risk $L^\star$, the optimal excess risk is $\widetilde{\Theta}(\sqrt{L^\star d_N/n}+d_{DS}/n)$, uniformly in the alphabet size, attained by a learner that knows neither $L^\star$ nor the confidence level. The upper bound composes the cover-menu-compression architecture of [CEH+26], at the realizable rate of [Pab26], with a new comparator-facing relative compression theorem: a size-$k$ compression rule that empirically dominates a comparator $h$ has population risk at most $L(h)+O(\sqrt{L(h)\Gamma}+\Gamma)$ with $\Gamma=(k\log n+\log(1/\delta))/n$, without stability; this transfers the comparison principle of the sharp binary theory [MQZ26] while discarding its Boolean-cube geometry, which does not lift to multiclass labels. The lower bound forces both terms using one class and one distribution at every fixed $L^\star$, by a pair-Assouad scheme calibrated to $L^\star$ and a fiber argument on the pseudo-cubes underlying the Natarajan-versus-DS separation of [BCD+22]. Both theorems extend to list learning: against the best $r$-tuple of hypotheses, the same architecture and the same two engines yield an optimistic rate and a lower bound of the same shape, forcing the fluctuation term that [Pab26] expected to be necessary against list comparators, and removing the factor $r$ from the known realizable list lower bound.

An Efficient Near-Optimal Algorithm for Adversarial $m$-Set Bandits

We study adversarial combinatorial bandits with $m$-set actions, where at each round the learner selects $m$ out of $d$ items and observes only the aggregate loss of the selected items. The resulting action set contains $K=\binom{d}{m}$ elements and can therefore be exponentially large. Nevertheless, the loss of every action is determined by the same $d$-dimensional vector of item losses. We propose a computationally efficient algorithm that exploits this structure without explicitly enumerating the action set. Against adaptive non-anticipating adversaries, it guarantees, with probability at least $1-\delta$, regret against the best fixed action of \[ R_T = O\left(\sqrt{dT\log(K/\delta)}\right). \] This matches the high-probability regret bound of the finite-action EXP3-KW algorithm of Zimmert and Lattimore, whose direct implementation may require exponential space. Our algorithm instead represents each sampling distribution with $d$ parameters and runs in polynomial time without enumerating the action set. Thus, it resolves the open problem posed by Maiti et al.

The Price of Fairness in Active Learning: Fundamental Limits and Optimal Label Acquisition

When a classifier is trained by actively choosing which examples to label, should its labeling budget depend on which notion of fairness it must satisfy? We answer this with the first information-theoretic lower bounds for group-fair active learning, and the answer is a sharp hierarchy across the three standard notions. Demographic Parity (DP), which equalizes the positive prediction rate across groups, can be verified from unlabeled data alone, so it adds no labels beyond ordinary learning. Equal Opportunity (EO), which equalizes each group's true positive rate, is fundamentally harder: estimating a rate conditioned on the true label forces the learner to collect labeled positives from every group, an irreducible cost that no query strategy can avoid. Equalized Odds (EqOdds), which additionally equalizes false positive rates, costs no more than EO whenever positives are the minority, the typical regime. This exposes a separation at the heart of fair learning: active querying yields the well-known exponential savings for accuracy, but no speedup whatsoever for fairness verification. Concretely, certifying EO to fairness tolerance γ requires Ω(d/ε2 + k/(ε2 p+)) labels, where d is the model's Vapnik--Chervonenkis (VC) dimension, k the number of groups, ε the target error, and p+ = Pr[Y=1] the positive rate; the fairness term is independent of ε, so it is a floor that active learning cannot lower. We give matching algorithms (ConstrainedERM-DP for DP, FairStratified for EO and EqOdds) that are tight up to logarithmic factors. The cost is largest exactly when positives are rare: at p+ = 0.01, as in fraud detection, EO demands roughly 90× more labels than DP. Experiments on Folktables, COMPAS, and Adult confirm the predicted scaling and yield 36 to 39% label savings over passive fair-learning baselines.

Optimal-Dimension U-Calibration by Bayesian Bootstrap