Skip to content

The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering

Aug 2026 · arXiv.org · Vol abs/2608.21262 · 2 citations · 115 references
Computer Science Mathematics

Abstract

Suppose we want a cutoff that 90% of a population falls below. We estimate it from a sample, and another sample would give a different cutoff and a different fraction below it. We ask how much that fraction varies when observations come in independent groups, such as pupils in classrooms or sentences in news articles. We prove that grouping multiplies its large-sample variance by $1+(m-1)\rho_I(p)$, where $m$ is the group size, $p$ is the target fraction, and $\rho_I(p)$ measures whether two members of a group fall on the same side of the cutoff. That correlation can differ from the correlation between the scores themselves, and it changes with the target. We give a direct proof, a counterexample to using score correlation, and an extension to unequal group sizes. A dataset therefore does not have one effective sample size. How much information it contains depends on the question you ask. In our document experiment, the same 1,000 rows carried about 217 independent observations'worth of information at the median. At the 95th percentile, they carried about 621. Nothing about the dataset changed. We asked it a different question. The number of rows is a property of the dataset. The effective sample size belongs to the analysis.

View source

Similar papers

Preprint Nov 2026

Studentized inference in randomized experiments when a few units dominate the outcome variation

In completely randomized experiments with heavy-tailed outcomes, a few units can carry a share of the residual variation that does not vanish as the number of units grows. We study the studentized difference in means and Lin's regression-adjusted estimator, with covariates whose leverage vanishes, when the finite popul...

Xin-Yuan Chen, Jonathan W. Kiesel, J. Woody · 0 citations
Preprint Oct 2026

Comparing tail forecasts: the number of exceedances is the sample size

When two forecasts of a tail, such as prediction intervals at high coverage, extreme quantiles or expected shortfall (ES), are compared by a consistent score, the score difference is known in advance except on the days on which the outcome falls beyond a forecast. The number of these exceedances, not the number of obse...

Raphael Coelho · 0 citations
Preprint Oct 2026

The Reach of Treatment Effects: Randomization Inference for Zero-Effect Regions

A treatment may affect some individuals while leaving others unaffected. We study the reach of treatment effects by asking how large an unaffected covariate region could be. We develop efficient local randomization tests over nearest-neighbour regions to construct a finite-sample valid upper confidence bound for the si...

Peng-Yue Li, Yao-Jun Zhang · 0 citations

Wasserstein Causal Forests for Distribution-Valued Outcomes

This paper proposes Wasserstein Causal Forests (WCF) for settings in which each unit's outcome is itself a probability distribution. This study also defines finite-grid transformed average and conditional average treatment effects, including a reference-distance contrast that asks whether treatment moves unit-level dis...

H. G. Souto · 0 citations
Preprint Sep 2026

What Does a Benford Test Actually Test? Marginal Conformity, Sampling Structure, and Forensic Inference

Benford's law specifies a marginal distribution for significant digits, whereas the usual first-digit Pearson $p$-value is calibrated under an independent multinomial sampling model. We separate these statements with four constructions that share the same one-time or pooled Benford target but have different joint struc...

Arthur Charpentier · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.