Why Fisher's exact test is least exact where it matters most: achieved size, wasted power, and a routing rule for 2x2 tables -- m02d Reproducibility deposit

Abstract

Why Fisher's exact test is least exact where it matters most: achieved size, wasted power, and a routing rule for 2×2 tables William J. Dwyer, MD, MPH, FAAP — Department of Mathematics and Statistics, University of Massachusetts Lowell. ORCID 0009-0004-0855-7222. Concept DOI (always resolves to the latest published version): 10.5281/zenodo.22238930. This release (v1.1.19):10.5281/zenodo.22264918, published Sep 3, 2026; the concept DOI above resolves to it. The previous release, v1.1.14, is 10.5281/zenodo.22262453 (Sep 2, 2026); earlier, v1.1.13 is 10.5281/zenodo.22262451, v1.1.5 is 10.5281/zenodo.22260812 and v1.1.4 is 10.5281/zenodo.22260570. What this is The reproducibility deposit for the m02d paper. "Fisher's exact test" is exactly valid — its true rejection rate never exceeds the nominal α — but it is not size-exact: because the 2×2 reference distribution is discrete, no table boundary falls at α, so the achieved (unconditional) size sits below nominal. At small designs the test spends only 26–66% of its α budget, and wasted size is wasted power. The paper computes the achieved size, the wasted-power cost, and the significance-flip behaviour of Fisher's exact against a conservative→liberal roster (Yates χ², Fisher, mid-p, Boschloo, Barnard, Pearson χ², Cressie–Read λ=2/3, likelihood-ratio G), and distills the result into a routing rule for the 2×2. Everything is deterministic exhaustive enumeration — two independent binomials, no Monte Carlo, no random seed — so there is no simulation error: every number, table, and figure reproduces exactly from a named script. The keynote A real-data scan of 6,030 public 2×2 tables (757 pydataset/Rdatasets datasets) shows the tests agree essentially everywhere away from the threshold but disagreement spikes on each decision line — ~67% of tables within ±0.02 of 0.05 and ~76% within ±0.02 of the 0.10 trend line get a test-dependent verdict — so the tests are miscalibrated relative to one another exactly where we use them to draw the significant/not-significant line, and the trend line inherits the same spike. Real-data direction is read against the valid band — the deployable tests valid at every design {Yates χ², Fisher, Boschloo, Barnard} — not against any single "frontier": a test over-rejects when it rejects where no valid test does and under-rejects when it misses what all valid tests catch. Validity itself is a size property, judged from achieved size (never from one table), and is kept strictly separate from real-data direction; a §2 definitions block fixes this vocabulary (validity as a size property; the size-α oracle as a reference, not a deployable test; the valid band as a set with no single winner). What the deposit contains Manuscript (author + anonymized, .md), the Derivations companion (D1–D6: conditional validity and the achieved-size shortfall; mid-p as the mean of the median-unbiased rule with E₀[mid-p] = ½; Boschloo's uniform power dominance; the nesting that makes flips one-directional; the atom-lattice gap and exact Boschloo region behind Figure 3; and why conditioning forfeits the budget), the novelty / prior-art companion, and the cover letter(The American Statistician). Reproducibility apparatus (rerun/) — the exact-enumeration engine achieved_size.py (tie-aware conditional p-values, validated against SciPy to 2.2e-16) and the deterministic sweep drivers: power_and_flips.py, size_skew_sweep.py, flip_boundary_sweep.py, roster_sweep.py (the full comparator roster, including the Cressie–Read λ=2/3 power-divergence member), and public_flip_scan.py (the real-corpus scan). Later builds add the sixteen-statistic achieved-size-and-power engine comprehensive_roster.py, the full per-test α-budget-grid engine budget_grid_all.py with the vectorized Boschloo helper boschloo_fast.py, the size-α oracle and size-adjusted-power engine size_adjusted_roster.py (the randomized budget-spending reference and each test's valid, size-matched power), and reference_grids.py — the finer-than-deposited hi-resolution reference/lookup tables (Reference Tables R1–R5) that back Figures 4 and 5, alongside (not replacing) the coarser Table S3. Each writes its locked JSON/CSV. Figures (figures/) — the achieved-size and wasted-power panels, the size×skew heat maps, the α=0.05 flip-boundary maps, the χ²-inclusive roster figures, the real-data flip-incidence figure, the routing flowchart (now Figure 9), and the per-test direction butterfly (Figure 8). Interactive demonstrator honest_fisher.html — a self-contained in-browser calculator with the full comparator taxonomy and the live-highlight routing flowchart; its JavaScript engine reproduces the deposited numbers and self-checks a parity badge on load. Deep-dive record (deepdives/) — the supporting analyses, including the size/skew, flip-boundary, χ²-roster, real-data, and computed-clearance write-ups, and the M0u-bundle assessment that reconciles m02d with the author's earlier Monte-Carlo program (below). LICENSE, CITATION.cff, MANIFEST.txt (SHA-256 of every file). The Cressie–Read λ=2/3 reconciliation The roster now carries Cressie–Read λ=2/3, the best-calibrated member of the power-divergence family that also contains Pearson (λ=1) and the G-test (λ=0). Across R×C shapes it is the best-calibrated test in the author's companion Monte-Carlo study; on the pure 2×2 that is m02d's subject, its exact achieved size is 0.062 / 0.053 / 0.055 / 0.055 at the representative designs — still mildly liberal, because small-sample discreteness bites it too. So the paper names λ=2/3 as the calibrated member of the χ² family, scopes the "common χ² defaults overshoot" claim to Pearson/Yates/G, and shows that even the well-chosen asymptotic statistic does not escape the 2×2 discreteness the exact unconditional tests handle — turning a potential referee objection into a strengthening of the routing rule. The companion 18-test Monte-Carlo study independently confirms m02d's Boschloo/Barnard power gap over Fisher (+3 to +10 pp by simulation vs +3.7 to +11.1 pp by exact enumeration here). All computation is deterministic exact enumeration; code is released under the MIT License, and text, figures, and data under CC BY 4.0. The public 2×2 corpus is built from the Rdatasets collection via pydataset; no data are redistributed beyond the illustrative tables cited in the paper. How to cite Please cite this deposit if you use the package or the method. Citing the concept DOI references the work in general and always resolves to the latest version; cite a specific version DOI to point at an exact snapshot. Dwyer, W. J. (2026). Why Fisher's exact test is least exact where it matters most: achieved size, wasted power, and a routing rule for 2×2 tables — reproducibility deposit [Software]. Zenodo.https://doi.org/10.5281/zenodo.22238930 BibTeX: bibtex @software{dwyer_m02d_2026, author = {Dwyer, William J.}, title = {Why Fisher's exact test is least exact where it matters most: achieved size, wasted power, and a routing rule for 2x2 tables --- reproducibility deposit}, year = {2026}, publisher = {Zenodo}, doi = {10.5281/zenodo.22238930}, url = {https://doi.org/10.5281/zenodo.22238930}, orcid = {0009-0004-0855-7222} } The DOI above is the concept DOI (resolves to the latest version); to cite a specific release use that version's DOI in its place. When the accompanying journal article appears, please cite it as the primary reference for the method and this deposit as the reproducibility archive. Version history v1.1.19 — ✅ 10.5281/zenodo.22264918 (published 2026-09-03) (m02d_fishers_exact_not_exact_reproducibility_v1.1.19.zip, md5 5235e7db685e9212047cca1e43319612, 4,723,287 B, 107 files): hi-resolution reference tables for Figures 4 and 5, and figure resolution restored to 300 dpi. Adds five finer-than-deposited reference/lookup tables — R1 Fisher, R2 mid-p, R3 Boschloo (backing the Figure 4 α-budget comparison) and R4 Pearson χ², R5 G (LR) χ² (backing the Figure 5 achieved-size heat maps) — tabulating realized size to four decimals across n = 6…100 (every integer through n = 30, where the discreteness is sharpest) and p = 0.05…0.50, each with a worst-case sup₍ₚ₎ validity column. New engine reference_grids.py (+ JSON + the markdown block) computes them by exact enumeration; sup checks reproduce Table S1 (Fisher/Boschloo valid, Pearson liberal at 0.0768). The tables enter the SN supplement as a new §S4 (Computational details → §S5) and are appended to the TAS supplement; the coarse Table S3 and roster grids are retained unchanged for consistent table sizes and side-by-side comparison. Figures 4 and 5 are re-rendered at 300 dpi. No computed result changed. Deterministic (identical md5 on two runs). Supersedes the staged v1.1.15–v1.1.18. v1.1.18 — staged, superseded by v1.1.19 before upload (…_v1.1.18.zip, md5 342e34f1799fa9b14ea8cb9666118094, 5,820,011 B, 104 files): a transient build that raised Figures 4 and 5 to 4× pixel density; reverted to 300 dpi in v1.1.19 once the intent (larger reference tables, not larger figures) was clarified. No content change. v1.1.17 — staged, superseded by v1.1.19 before upload (…_v1.1.17.zip, md5 6bd5b6c3fb313fa6306c2f624553b3ae, 4,393,871 B, 104 files): the §2 definitions block. Adds a short "terms and reference objects" block at the end of Section 2 that fixes the vocabulary the rest of the paper depends on — validity as a size property (read from achieved size, never one table); the size-α oracle as a randomized budget-spending reference, not a deployable test (Tocher 1950; Lehmann and Romano 2005, both added to the reference list); and the deployable valid tests as aband, not a frontier (Boschloo one member, distinguished by dominating Fisher, not the oracle or the edge) — with a coda keeping validity and real-data direction strictly separate. Exposition over existing output; no computed result changed. Deterministic. v1.1.15 /