Three of five model and method pairings the authors ran collapse to chance MMLU at 1,000 sequential edits under published hyperparameters, while edit success stays at 1.00 and locality reads clean.
Benchmarks scored by an LLM judge routinely adjudicate differences of a tenth of a point, but the resolution of those benchmarks has never been measured. Existing sample-complexity work covers accuracy benchmarks and leaves the judged case open. Treating the system as the object of measurement, we decompose 373,019 jud...
This work reports 10,369 claims with 184 execution-verified flips mined from 23 Python libraries, splits held out by repository, a post-knowledge-cutoff split, a shuffled-diff null, a paraphrase control, and a leave-one-repository-out analysis over 17 repositories.
A training-free remedy: an instruction compiler that rewrites the stacked prompt in a single LLM call and is reused across queries is evaluated, recovering up to +11 points of follow rate for weaker models, while leaving stronger models, which already internalise the same structure, essentially unchanged.
Canary tools are introduced: diagnostic probe tools planted in an agent's Model Context Protocol (MCP) tool set, each engineered to probe one specific tool-selection weakness, evidence that the probes measure reasoning, not phrase-spotting.
Atul Anand, S. Chattaraj· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.