An auditability discipline built at production time is specified: git sealing with an anchor lineage, hash-bound provenance, red-line gates that refuse non-compliant artifacts and log every refusal, cross-model role separation, and programmatic assembly from registered sources.
Abstract
Language models now draft, classify and criticise inside research production, yet the artifacts they help produce carry little accountable history. Rather than detecting machine involvement afterwards, we specify an auditability discipline built at production time: git sealing with an anchor lineage, hash-bound provenance, red-line gates that refuse non-compliant artifacts and log every refusal, cross-model role separation, and programmatic assembly from registered sources. Adherence is instrumented by metric cards, each carrying a pre-registered blind spot and evidential standing, frozen before the prospective case it observes. In that case the observed project's pre-registered confirmatory test was executed under seal and returned No-Go, and that project's frozen stopping rule halted the work, against its own operators. A lower-graded retrospective case covers families whose machinery predates the protocol. Current observations are provisional; we release a package from which a third party can recompute every primary metric.
The case suggests that explicit failure records, fixtures, port-owned proofs, and validating imports can make agent-assisted systems more auditable and controlled ablations and external replications are needed to test whether such infrastructure causally improves development outcomes.
A pipeline promoting an AI system publishes records claiming the thing evaluated is the thing deployed and that the evidence licensed the transition, and measures whether those records can express that claim and whether it holds where declared.
F(AI)2R is FAIR research with AI in the loop, twice: an AI-assisted authoring pass and a machine-readable audit pass over every artefact. AI systems now draft, refactor, and verify research artefacts, yet their contributions are rarely recorded in a form a later human or machine can audit. Building on the original F(AI)2R experiment, we generalize its provenance model beyond scholarly writing into aiprov, a PROV-O extension covering any AI-in-the-loop artefact, and we package the method as an executable skill that an AI agent operates itself: setup asks the human operator for their ORCID ID, resolves their identity from the public registry, and scaffolds continuous integration that gates every push on graph conformance and publishes the current build of this very paper. The paper is its own case study. Every activity, claim, and source in its production is recorded in the repository's provenance graph under two invariants: no parentless claim, and verification rungs that only humans may grant.
Large language models (LLMs) can turn a flood of cross-layer industrial logs into a fluent incident narrative, but a narrative that cites only real, resolvable events can still be wrong in every relation that matters: the login came from a different workstation, the write command occurred after the physical change it supposedly caused, the action fell inside a planned maintenance window, and the controller does not even actuate the affected process. A cited event is not necessarily supporting evidence. When such a narrative drives automated response, the error propagates into isolating the wrong controller or revoking a legitimate operator. We present EviGuard, a system that decides when an LLM’s understanding is trustworthy enough to act on. EviGuard stores auditable cross-layer evidence in a provenance graph, lets the LLM propose only hypotheses, compiles each hypothesis into atomic machine-checkable claims in an Incident Claim Language, and has an ensemble of deterministic verifiers label every claim supported, contradicted, or unknown against the graph—honoring interval time, event-time policy and credential versions, network reachability, and physical control dependencies. A response gate forbids any high-impact action whose critical preconditions are not all supported. On EviCPS-Bench (42 hardware-in-the-loop attack chains, 9600 claim-level labels, κ=0.87), EviGuard cuts the unsupported-claim rate from 12.6% to 1.7%, raises relation-edge F1 from 0.64 to 0.89, holds prompt-injection success to 0.4%, and executes zero unverified high-impact actions across 3200 response decisions, at a median end-to-end latency of 0.44 s.
Hao-Zhe Zhou, Hang Lei, Mao-Lin Yang· Applied Sciences· 0 citations
The results support verifier-selected supervision for bounded, machine-checkable capabilities, not arbitrary planning, enterprise validity, or unrestricted self-improvement.
Contract-clause extraction benchmarks measure reference recovery, but interpreting a system for review assistance also requires measuring unnecessary output and the evidence available within categories. This paper presents a retrospective, reproducible evaluation of a fixed public RoBERTa checkpoint on CUAD's published 102-contract test split. Candidate generation is preserved from an earlier frozen run; matching errors are corrected and category thresholds are reselected on 62 training-split contracts using the original 90% recall target. The corrected operating point recovers 82.2% of reference spans at 8.2% CUAD precision. Of 30,464 returned candidate strings, 19,562 occur on questions with no annotated answer. The system answers 1,335 of 2,938 such questions, a 45.4% false-positive rate (95% contract-bootstrap interval: 43.3-47.9%). Matching candidates constitute 20.3% of returned strings, illustrating why reference-based precision and candidate-level review burden require different denominators. Only nine of 41 categories have at least 30 positive and 30 negative contracts; this count changes to 18 and two under support minima of 20 and 40. Two categories have no negative contracts, making their no-answer false-positive rates undefined. These findings describe one checkpoint, decoder, and threshold policy. Earlier test access and uncertain checkpoint training overlap limit confirmatory interpretation. The paper supplies the original and corrected analyses, predictions, and software checks as ancillary artifacts; it establishes neither a new release criterion nor production readiness.
Zeki Emre Tekin· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
