The Model Is a Dependency: Provenance and Error Detection in Financial Data Extraction

Abstract

This work was produced through an AI-assisted workflow directed and reviewed by the author. This study separates identifying a model run from checking whether its numeric answer is correct. It evaluates a membership-only source check and top-20 truncated token entropy on a small water-table task, then examines extraction from 25 SEC compensation filings under five text presentations. The membership check flags none of 106 stored misattributions; this follows from defining that error class as a wrong value whose digits occur in the source. Entropy discrimination varies by model. The tested percentile thresholds miss errors in some models, but the experiments do not establish that all transferable thresholds or other audit mechanisms must fail. No closed model was tested. Within the selected filings, preserving an aligned grid reduces misattribution for the tested models. Filing-level resampling describes uncertainty within that non-random sample; it does not establish an ordering in an unobserved deployment population. The corrected negative-answer subset contains 45 sign errors in 598 repeated calls covering 13 questions from 9 filings. A source-sign exception corrects seven labels and six verdicts, leaving every stored answer and misattribution indicator unchanged. Both Qwen quantisation builds now have 472 correct answers out of 472 in the paired grid-intact condition, leaving no observed error difference there. The retained results support selected arithmetic checks and a narrower provenance requirement: weights are one component of a run that also depends on prompts, configuration, runtime and sources. The released records do not establish complete historical replay. The study therefore provides bounded evidence about two checks and these extraction conditions, rather than a general verdict on open versus closed models. The research design, method, sourcing decisions and analytical judgements are the author's. An Anthropic model assisted the original experiment, scripts and drafting. OpenAI GPT-6 assisted the source checks, calculations, code repairs and manuscript revision, so this text was artificially generated and was reviewed by the author before publication. The assisting models are not always neutral parties to the subject matter. Anthropic and OpenAI compete with providers discussed; OpenAI's closed-interface interests are relevant to the framing. Google Gemini supplied separate-model review of bounded corrections and has a direct Gemma interest in five corrected verdicts. No closed model was tested. Author research and publication decisions remain separate from model advice. Guarantee: every retained measured result is traced to the primary record named in the Verification note. paper_tables.py regenerates the paper's six empirical tables from the supplied datasets. Modelled or estimated inputs are identified as such. A reader can check this without trusting either party. The guarantee concerns stored-output calculations and the documented source exception, not complete historical replay or source truth for every row. REPRODUCTION.md lists the retained datasets and additional checks. Unverified historical claims are labelled and excluded from retained findings. What the author cannot guarantee: a language model's output can be wrong in ways that survive review. In prose the error is locally plausible and consistent in tone with what surrounds it; in code it simply runs, and a wrong constant or a mis-set filter still returns a clean number. Several methods have been deployed to mitigate this, including explicit instructions, internal red-teaming and cross-lab blindspot checks, but the author does not claim the review is exhaustive. Corrections are logged against the DOI when surfaced. No warranty is offered beyond the terms of the CC BY 4.0 licence. Independent analysis and open-science documentation only.