A near-universal test of independence: the three-pole calibrated envelope, its joint null law, and dependent-null higher-criticism attainment — (m02f) reproducibility deposit
Abstract
A near-universal test of independence: the three-pole calibrated envelope, its joint null law, and dependent-null higher-criticism attainment William J. Dwyer, MD, MPH, FAAP — Department of Mathematics and Statistics, University of Massachusetts Lowell. ORCID 0009-0004-0855-7222. Concept DOI (always resolves to the latest version): [10.5281/zenodo.XXXXXXXX] (fill from the record's "Cite all versions" DOI — a distinct number from the version DOI below; once supplied, this line, the How-to-cite block, and the manuscript availability line are finalized). What this is The reproducibility deposit for the three-pole calibrated envelope D3, a near-universal test of independence for an R by C contingency table. No single statistic is optimal along the sparse-to-dense axis: a quadratic omnibus (Pearson Q) is efficient against dense departures but sub-optimal against sparse ones, higher criticism (HC) attains the sparse detection boundary over moderate sparsity, and a maximum-cell rule (M) dominates in the very-sparse strong corner. D3 rejects when the minimum of the three self-calibrated p-values is small, the minimum recalibrated to its own null; it lands within about 0.09 of the best single test everywhere on the axis. The reference law is a new distributional object, the joint null law L_N of (Q, HC, M) under the exact margin-conditional null and of their minimum-p-value envelope. Sole author: William J. Dwyer. Sibling of the detection-boundary companion (arXiv:2608.07979). What the deposit contains Manuscript and a long-form derivations companion (Theorem 1 with Corollary 1.1; Theorem 2; and the D3.9-D3.10 short-range certificate) plus the OUP imaiai LaTeX source and article-class proof PDFs/DOCX, covering: the closed-form envelope tail slope d_eff = 2 - lambda_Q and the trace-gamma x Gumbel coupling with its closed-form min-p law P(p_min <= t) = 1 - (1 - t)[1 - (1 - lambda_Q)t]; the covariance lemma and boundary attainment; the very-sparse Berman closure and the moderately-sparse closure by a DIRECT Gaussian comparison (the residual field is exactly Gaussian, so no functional CLT is needed; the CCK comparison error is the threshold-process covariance difference, exactly O(r_N) with no log growth), which makes Theorem 2 PROVED UNCONDITIONALLY on its stated near-square scope (min(R,C) >= N^delta, both dimensions growing), only a sub-polynomial-min regime staying conditional on the stated slowly-varying-kernel lemma; and the exact short-range certificate (correlation row sum identically 4, signed row sum identically 0, spectral norm tending to 1) that shows the fixed-margin field is never Hall-Jin adversarial. The two-pole collapse (lambda_HM -> 1, lambda_Q -> 0) is proved (Darling-Erdos edge + Poisson extremes; Li-Xue 2015 asymptotic independence). Interactive demonstrator honest_detection_d3.html — the three-pole routing and calibrated-envelope verdict in the browser. Reproducibility scripts — every reported number traces to a named, deterministically-seeded script (seed 20260909): leads_followup_A.py (near-universality), minp_tail_slope.py / minp_envelope_law.py (Theorem 1 / Corollary 1.1), hc_covariance_lemma.py / hc_conditional_achievability.py / hc_berman_gumbel.py (Theorem 2), hc_modsparse_covariance_route.py / hc_modsparse_equicontinuity.py (moderately-sparse closure), and the bounded-dimension / short-range-certificate engines hc_boundeddim_corner.py, hc_twosample_companion.py, hc_twosample_companion_v2.py, hc_midp_calibration.py, hc_randomized_pvalue.py, hc_shortrange_and_exponent.py, hc_exponent_hires.py. Findings ledgers — the prior-art / red-team findings ledger and the bounded-dimension / closure deep dives (the 2 by C corner reduces to the two-sample frequency-table problem; its apparent obstruction is p-value atomicity, resolved by exact-discrete / randomized calibration). Figures and the deterministic deposit builder (fixed timestamps → stable md5). All evaluation is simulation-based and deterministic given the seed. Code is released under the MIT License; text, figures, and data under CC BY 4.0. How to cite Please cite this deposit if you use the package or the method. Citing the concept DOI references the work in general and always resolves to the latest version; cite a specific version DOI to point at an exact snapshot. Dwyer, W. J. (2026). A near-universal test of independence: the three-pole calibrated envelope, its joint null law, and dependent-null higher-criticism attainment — reproducibility deposit [Software]. Zenodo.https://doi.org/10.5281/zenodo.XXXXXXXX BibTeX: bibtex @software{dwyer_m02f_2026, author = {Dwyer, William J.}, title = {A near-universal test of independence: the three-pole calibrated envelope, its joint null law, and dependent-null higher-criticism attainment --- reproducibility deposit}, year = {2026}, publisher = {Zenodo}, doi = {10.5281/zenodo.XXXXXXXX}, url = {https://doi.org/10.5281/zenodo.XXXXXXXX}, orcid = {0009-0004-0855-7222} } The DOI above is the concept DOI (resolves to the latest version); to cite a specific release, use that version's DOI and add version = {v1.0.NN}. When the accompanying journal article (Information and Inference, sibling of arXiv:2608.07979) is published, please cite it as the primary reference for the method and this deposit as the reproducibility archive. Version history (consolidated changelog) Published version DOIs are marked ✅; the concept DOI above always resolves to the latest. Staged versions were rolled into the next published one unless noted. v1.0.7 ✅ 10.5281/zenodo.22681615 (2026-09-09; PUBLISHED) — Rung-4 more-data-and-leads pass: reduces the single remaining conditional step to one universal scalar and measures it with signal across a wide shape variety; no prior computed number, datum, figure, or manuscript claim changed. New seeded engine hc_rung4_edge.py establishes the reduction Rung 4': expanding each family kernel about the iid background gives B(t) = kappa'(0) t S_shape + O(r_N^2), so Rung 4 collapses from an N-dimensional bound to bounding one universal scalar kappa'(0) = dkappa_pair/drho|_0 — and c = 0 holds iff kappa'(0) = o(min/N) (for square, o(N^{-1/2})). Part A measures kappa'(0) by a CRN first-difference dE[F]/drho: it is universal across pair positions (near vs far agree within CI at every N) and decays like N^{-0.97} — polynomially faster than the N^{-1/2} threshold. Part B shows the anti-concentration modulus Q(HC+) is flat in N (~N^{+0.04}, O(1), inside Nazarov's sqrt(log N)). Part C confirms the first-order prediction matches the direct measurement across aspect 1–16 and thin 3x48 (not a near-square artifact). Deep dive Dwyer_M02f_Rung4_EdgeLeads_2026-09-09 gives four discharge routes (Nazarov anti-concentration; CCK-Koike 2022 sharpened comparison; second-order Gaussian Poincare; Mehler closed-form single-pair kernel). Honest verdict: c = 0 is still NOT unconditional — Rung 4 is sharply isolated to a scalar edge estimate with a polynomial margin and four routes, but not discharged verbatim — so Theorem 2's status is unchanged (near-square unconditional; c = 0conditional). Deterministic m02f_near_universal_independence_v1.0.7.zip, md5 f3300e3d031f102061b5bea73f3e68a4, 112 entries (member diff vs v1.0.6: +4 — engine .py, its CSV, and the deep dive .md+.pdf; guard manifest updated to route the new note; all prior code/data/figures and the manuscript/arXiv sources byte-identical). v1.0.6 (2026-09-09; staged locally; version DOI pending upload) — standalone slowly-varying-kernel NOTE: a rung-by-rung proof attempt (no prior computed number, datum, figure, or manuscript claim changed).Dwyer_M02f_SlowlyVaryingKernel_Note fills the lemma-proof slot the submission guard reserved. It PROVES Rung 1(row/column exchangeability + permutation-invariance ⇒ the smart-path kernel kappa_ab = dE[F]/dSigma_ab is EXACTLY family-constant, so the family sum collapses onto the signed identity: E[F(z)]-E[F(g)] = (1/2) integral N B(t) dt, B = -kR-kC+kB), Rung 2 (Price/Stein representation), and Rung 3a (the zeroth-order kernel vanishes, kappa_0 = B(0) = 0, by count-process separability + sign symmetry); and REDUCES the remainder (Rung 3b/4: B(t) = O(r_N) × edge factor) to a single Nazarov/CCK anti-concentration estimate at the Darling-Erdos edge, with two independent cross-checks (second-order Poincare on the exact sum rho^2 = o(N); signed-vs-unsigned aggregate). New seeded enginehc_kernel_path.py measures the kernel along the smart path: kappa_0 consistent with 0 at t = 0, and the integrand N B(t)consistent with 0 across t in {0,0.5,1} at 16x16 and 24x24. Honest verdict: c = 0 is NOT made unconditional — the last estimate is set up, not discharged verbatim — so Theorem 2's status is unchanged (near-square unconditional; c = 0 conditional on Rung 4). D3.10 records the four-rung decomposition. Deterministic m02f_near_universal_independence_v1.0.6.zip, md5 1151e01aa76657acc750d129348e89bb, 108 entries (member diff vs v1.0.5: +4 — engine .py, its CSV, and the note .md+.pdf; derivations rebuilt to cite the note; all prior code/data/figures otherwise byte-identical). v1.0.5 (2026-09-09; staged locally; version DOI pending upload) — high-resolution decomposition of the one remaining conditional item (the slowly-varying-kernel lemma / c=0 outcome) into ranked proof leads; no prior computed number, datum, figure, or manuscript claim changed. New seeded enginehc_kernel_lemma_leads.py measures the smart-path kernel kappa_ab = E[d2_ab F] = dE[F]/dSigma_ab (Price identity) with 95% CIs and decomposes it: within-family kernel differences consistent with 0 (14/15 checks) — the numerical face of an EXACT permutation-symmetry fact that kappa is family-constant, reducing "slowly varying across the partner index" to exact within-family constancy (LEAD 1); the leading transfer term E1 = (N/2)(-kR-kC+kB)statistically 0 at every square shape and not grow
