Skip to content
Preprint

When a High Score Is an Illusion: Certifying Genuine versus Repackaged Forecasting Skill

Sep 2026 · 0 citations · 42 references
Mathematics

TL;DR

This work characterize assignments that preserve association between specified population-rank contrasts, including forecast rank minus baseline rank compared with outcome rank minus baseline rank, including forecast rank minus baseline rank compared with outcome rank minus baseline rank.

Abstract

Ranks depend on the observations used for comparison. Reusing those observations can add association between forecast and outcome rank contrasts even when the evaluated forecast and outcome stay fixed. We characterize assignments that preserve association between specified population-rank contrasts, including forecast rank minus baseline rank compared with outcome rank minus baseline rank. Conditional on independent training, whole trajectories are sampled independently from a common law, with unrestricted dependence within each trajectory. The expected score separates into its target and an explicit interaction between map pairs. When reassigning references, we keep the learned maps, reference law and coefficient row sums fixed. Zero weighted reference overlap for every map pair is necessary and sufficient for preservation uniformly over permitted maps and laws. An unbiased three-trajectory kernel estimates the interaction; independent evaluation and validation provide finite-sample lower bounds. Complete U-statistics estimate the same target directly when all draws can be recombined. Sharp shared-baseline ranges, including ties, tighten both constructions. In a Beijing air-quality archive, interaction accounts for 91.4% to 94.5% of seven learned forecasts'expected shared scores under the empirical archive law. Separate results address category-fitting error and temporal feedback. In a matched category-control null experiment, distinct references reduce rejections from 402 to 41 out of 1,000 panels.

View source

Similar papers

How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?

LLM leaderboard gains can reflect selection among privately evaluated model variants, yet neither the number of variants nor their dependence is public. We ask how many hidden variants a published margin can support while retaining statistical evidence of a provider's advantage over a fixed comparator. For a fixed cand...

Chen Yang, Jun Chen · 0 citations
Preprint Sep 2026

Testing Conditional Stochastic Dominance via Copula Derivatives

Comparing two populations at the same physical covariate value requires more than conditional means or isolated target-point decisions: researchers may need evidence about an entire conditional-distribution ordering over a continuum, even when covariate margins differ. This paper makes that common-value comparison esti...

Wei-Qi Yang, Wei-Wei Zhuang, Xiao-Jun Song · 0 citations

A Free Knob: Decoupling Calibration and Predictive Skill in Threshold-Based Evaluation

A post-hoc transform fitted on held-out data and applied separately to each system is repurpose classical monotone calibration as a symmetric audit: a post-hoc transform fitted on held-out data and applied separately to each system.

Md Tanveer Hossain Munim, Bijoy Ahmed Saiem, Al-Amin Sany et al. · 0 citations
Preprint Sep 2026

Bayesian Confidence Recalibration and Research-Equilibrium Criticality: Temporal Support in Robust Portfolios

Robust portfolio rules that reconstruct confidence sets after learning need not preserve the evaluator obtained by prior-by-prior Bayesian transport. In the Gaussian model, this discrepancy is summarized by natural-coordinate displacement: inherited transport preserves it whereas fresh reconstruction can replace it. We...

Han Yanç · 0 citations

Wasserstein Causal Forests for Distribution-Valued Outcomes

This paper proposes Wasserstein Causal Forests (WCF) for settings in which each unit's outcome is itself a probability distribution. This study also defines finite-grid transformed average and conditional average treatment effects, including a reference-distance contrast that asks whether treatment moves unit-level dis...

H. G. Souto · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.