PDFa11yMut: Measuring Mutation-Specific Detection in PDF Accessibility Checkers

Abstract

Automated PDF accessibility checkers provide useful conformance evidence, but a clean report is not a complete accessibility oracle. PDFa11yMut measures mutation-specific checker behavior by applying paired structure-level transformations to reference-suite baselines, verifying intended deltas and non-target invariants, and recording hash-linked checker evidence. Across 30 conformance-oriented mutants, PAC and veraPDF each produced 30 and 30 direct target findings, respectively, while Acrobat produced 23 direct findings plus 3 prespecified consequence-proxy findings. The 39 semantic/assistive-representation mutants produced no automated target finding in the tested configurations, while Acrobat issued manual-review prompts for a subset; Class B is interpreted descriptively rather than as a universal checker obligation. A separate convenience-selected exploratory AT sample observed representation differences in 8 of 9 pairs under one fixed NVDA/Acrobat/Windows procedure. The artifact contributes reusable operators, structural and purity oracles, paired baseline/mutant evidence, and reproducible analysis for a scoped mutation-testing study rather than a general checker-accuracy benchmark.