Reproducibility archive for "Bounding Language-Model Failure Attribution at Industrial Evidence Interfaces"

Abstract

A large language model (LLM) running inside an industrial instrument will fail on some of what the operator asks. Whether such a failure is justified as evidence of a capability deficit of the model, or only of a burden the instrument left at its evidence interface, is not answerable from an accuracy score, and a deployed instrument has no continuously available one because its routine answers are not independently labelled. We state and test a bounded principle of deployment-level identifiability for that question: an observed failure is evidence of model incapability only relative to a declared family of interventions at the evidence interface, and what remains under that family is an upper bound on what may be attributed to the model, never the attribution itself. It is bounded because it identifies nothing outside the family it names, and testable because a running instrument can switch that family. The test is a full factorial over three deterministic mechanisms (compiling derived quantities into the evidence, declaring unknowns as explicit state, and validating typed claims) on a deployed phased-array ultrasound inspection instrument, scored into four failure classes, two of them without reference answers. The mechanisms remove 24.9, 22.5 and 18.9 points of failure from three models that fit an eight-gigabyte industrial GPU. Repeating the factorial on a second quantity space, compiled from a public machine-condition dataset with no detector retuned, four of the twelve factor-class effects keep one sign while the validator's net effect on accuracy reverses, from +2.5 points to -6.1. The same decomposition remains operational and discriminating in both; individual factor-class effects do not transfer uniformly. This record accompanies a manuscript under double-anonymous review. The authors are withheld and the archive is sanitised accordingly: the project name, the institution and every personal address are replaced with placeholders, and the title page and unmasked instrument captures are omitted. No number, script or datum is altered. A new version carrying the byline and the unredacted tree will be published on acceptance, under the same concept DOI. Reproducibility bundle: the evidence compiler, question generator, factorial runner, scorer and analysis, with the frozen configurations, the scored responses, the preregistration and the source data behind every plotted value.