Invertible Logits Transformation (InvLT), which applies a learned scalar MLP element-wise to the pre-softmax logits, consistently outperforms a broad set of post-hoc baselines on standard calibration metrics.
Abstract
Post-hoc calibration aligns a classifier's predicted confidences with its empirical accuracy without retraining. An ideal calibrator should correct nonlinear miscalibration, scale gracefully to large label spaces, and preserve the original predictions; existing methods typically violate at least one of these properties---temperature scaling lacks expressivity, more flexible parametric alternatives introduce parameters that grow with the number of classes $C$, and other expressive methods do not preserve the rank ordering of class scores and may alter the predicted class. We propose \textbf{Invertible Logits Transformation (InvLT)}, which applies a learned scalar MLP $f:\mathbb{R}\to\mathbb{R}$ element-wise to the pre-softmax logits. Sharing $f$ across all logit dimensions makes the parameter count independent of $C$. Monotonicity of $f$---and hence preservation of the argmax prediction---is softly encouraged via a paired inverse network rather than enforced through the numerical integration required by prior monotone calibrators; this avoids their computational overhead while empirically preserving the original classification accuracy in every setting we evaluate. Across standard image classification benchmarks and a range of architectures, InvLT consistently outperforms a broad set of post-hoc baselines on standard calibration metrics.
Outcomes are increasingly regressed on a calibrated probability vector for unobserved class membership, and that vector is often coarsened to a hard label first. Under a constant-coefficient structural mean and conditional calibration, the observed-data problem is a partially linear regression of the outcome on the probability vector; we take this reduction as the starting point and ask what coarsening costs. For any coarsening, the plug-in estimator converges to $\mathcal{A}\tau$, where the coarsening operator satisfies $\mathcal{A}=I+D^{-1}\mathbb{E}[a_{h}u^{\top}]$ with $u$ the discarded signal. Coarsening is therefore free exactly when what is discarded is uncorrelated with what is kept, and is otherwise anisotropic: it distorts some contrasts far more than others. The same operator governs inference. The Wald interval built from coarsened labels has limiting coverage $\Phi(z-\lambda)-\Phi(-z-\lambda)$, with $\lambda$ the ratio of the coarsening bias to the reported standard error; because $\mathcal{A}$ and that standard error depend on observables alone, the coverage implied by the estimated index can be approximated before the interval is reported. Simulations show severe coverage loss after argmax coarsening, and three real-data audits exhibit the direction-specific distortion that hard labels induce.
Classical likelihood-ratio tests and $\Delta$AIC exacerbate the statistical significance crisis by scaling with sample size, often flagging negligible improvements as highly significant. While causal estimands like the average treatment effect (ATE) quantify practical magnitude, their reliance on the expectation operator ties them to the data's original coordinate scale. Furthermore, existing pseudo-$R^2$ metrics are inadequate: variance-based measures ignore higher-order distributional changes, and current formulations lack invariance to monotone transformations. We resolve these limitations by introducing Entropic Variance (EV) as a rigorous, scale-independent generalization of error variance in ordinary least squares. We define the population EV-based parameter, $\rho^2_V$, which projects unbounded cross-entropy onto a standardized $[0,1]$ scale, and establish that the EV-based $F_\text{V}$ statistic asymptotically follows an $F$-distribution. Building on these distributional properties, we propose two estimators: the empirical population $R^2_{\text{SV}}$ and the out-of-sample predictive $R^2_{\text{SVP}}$. Both are derived by exponentiating per-observation cross-entropy and incorporate a degrees-of-freedom correction for training optimism. Leveraging the $F_\text{V}$-distribution, we derive refined $p$-values and confidence intervals for $\rho^2_V$ without requiring intractable Fisher information matrices. Simulation studies and a Parkinson's disease microbiome application demonstrate the superiority of variable selection via these EV-$R^2$ metrics. Notably, evaluating the $R^2_{\text{SVP}}$ of a LASSO path via data-splitting reduced false discovery rates from 80% to 6% in simulations while fully preserving signal recall.
Predictively oriented (PrO) inference quantifies uncertainty by selecting a distribution over model parameters to optimize a scoring rule applied to the induced predictive distribution, together with a divergence penalty from a reference distribution. By applying the scoring rule after averaging model densities, PrO inference targets predictive performance, accounting for model misspecification. We focus on the logarithmic score with general $\phi$-divergence regularization. Our contributions are twofold. First, we derive a finite-dimensional dual formulation of PrO inference. For $n$ observations, the dual problem has $n+1$ variables. We establish zero-duality-gap criteria and optimality conditions that relate the primal and dual solutions. When primal and dual solutions exist, these conditions yield a semi-analytical representation of the PrO posterior and certificates for assessing the accuracy of numerical solutions. For Kullback--Leibler regularization, the posterior has an exponential form. Second, we derive a finite-sample excess predictive-risk bound for approximate PrO posteriors that separates sampling fluctuation, approximation under a divergence budget, regularization, and numerical optimization error. The result applies even when the benchmark predictive risk is not attained by any probability distribution over the model parameters having finite divergence from the reference distribution. We use an exactly solvable categorical example to show that predictive-risk convergence can imply convergence to a unique predictive distribution even though the parameter distributions have no weak limit on the original parameter space. The example also shows that different $\phi$-divergences can require different regularization schedules. We conclude with a misspecified Gaussian location-mixture example that illustrates the dual computation, primal recovery, and numerical accuracy checks.
Aurya Javeed, D. Kouri, Teresa Portone et al.· 0 citations
When sensitive attributes are continuous and high-dimensional $-$ demographic score vectors, posteriors over attributes, age or income profiles $-$ enforcing full statistical independence is often too restrictive, and existing relaxations rely on indirect dependence penalties or adversarial schemes that do not directly target the fairness-accuracy trade-off. We instead consider mean demographic parity through DPVar, the variance of the conditional-mean prediction given the sensitive attribute, and show that optimizing it yields a functional bilevel problem. We propose two algorithms for this problem: FBO, which uses a closed-form adjoint we derive for the squared-loss case to obtain an exact hypergradient, and ITD, which differentiates through unrolled inner steps and extends beyond squared loss. On synthetic data and a new semi-synthetic benchmark built from 60 tabular regression datasets, both methods achieve the lowest or near-lowest aggregate fairness-accuracy regret, and consistently match or outperform strong HSIC, adversarial, linear-dependence, and generalized-DP baselines.
Ieva Petrulionyte, Julien Mairal, Michael Arbel· 0 citations
Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise. Because such judges prefer verbose or well-formatted answers, the natural fix is to add bias covariates to a Bradley--Terry model and estimate the bias away. We show this cannot work as advertised: the quality/bias split is \emph{not identified} by pairwise comparisons, and the failure is exact -- across $48$ real judge-pools the profile likelihood over the coefficient is flat to $\mathbf{0.0000}$ \textbf{nats}, and scaling the comparisons $26\times$ buys none. A ``debiased''score is selected by the prior, not recovered from data. Our contribution is accordingly not a better estimator but a characterization of \emph{when prior-based correction is justified}, plus designs that supply the missing information when it is not. The assumption the prior encodes -- quality is a priori uncorrelated with the covariate -- pays only while $\mathrm{corr}(\theta,x)$ stays below a crossing point (configuration-dependent, $0.22$--$0.60$), which is what makes the same model help on LLMBar and hurt on SummEval and Nectar. We give two escapes: a \textbf{trusted-anchor gate} that decides per (judge, covariate, task) (no false enables in $6{,}000$ decisions at $K\ge10$ anchors, a rate our sample bounds at $\le6\%$), and a \textbf{paired rendering design}. Across fifteen real LLM judges bias is heterogeneous and capability-dependent: correction improves \topk{} recall by $0.20$--$0.32$ on five biased-but-competent cheap judges and is a no-op on frontier ones (Spearman $\rho{=}{-}0.84$ between competence and gain over the $14$ competent judges, $p{<}10^{-3}$), concentrating the benefit where at-scale evaluation happens.
Jian Xu, Delu Zeng, John W. Paisley et al.· 0 citations