Skip to content

Where Did the Update Go? Certifying Declared Change Envelopes for Model Updates

Nov 2026 · 0 citations · 79 references
Computer Science

TL;DR

Leakage compares two models, so it can be certified from unlabeled traffic with anytime-valid tests whose sample cost the authors bound from above and below, and per-update certificates compose into a bound on lifetime drift outside all declared scopes.

Abstract

Deployed language models are updated continually, and regulators increasingly accept changes that were declared in advance. Showing that an update changed behavior only where it was declared to is a different statistical problem from bounding how much it changed: a useful update is meant to change some decisions, so global disagreement and regression bounds cannot certify it. We formalize the problem with declared change envelopes. Before training, the developer commits to a region of inputs, and every decision that changes outside it counts as leakage. Leakage compares two models, so it can be certified from unlabeled traffic with anytime-valid tests whose sample cost we bound from above and below, and per-update certificates compose into a bound on lifetime drift outside all declared scopes. We measure leakage for LoRA and full fine-tuning, preference optimization, reinforcement learning, and knowledge editing on models from 1.5B to 72B parameters. Narrow updates leak widely: domain and policy LoRA updates keep only about 7-36% of the decisions they change inside the declared scope. Across 256 trained variants, leakage tracks the update's KL divergence to the incumbent on out-of-scope inputs (Spearman 0.94). None of the customary remedies we tested makes a rule-like update certifiable without giving up much of its intended change. Distilling the incumbent's decisions on unlabeled prompts in the deployment format does: anchored LoRA, GRPO, and (at a small learning rate) full fine-tuning updates certify 5% leakage from a few hundred to a few thousand unlabeled samples, and the certificates replicate on held-out traffic and on written answers. The certificates cover decision-like outputs only; on open-ended chat, re-batching alone changes 11-21% of responses and no update certifies. Code: https://github.com/Usama1002/declared-change-envelopes

View source

Similar papers

Preprint Oct 2026

Beyond Update Counts: A Control Map for Evaluating Test-Time Model Updates

Test-time updating lets a language model change persistent state after training---such as model weights, adapter parameters, or a memory module---before or while answering queries. Existing evaluations often compare systems using update count or frequency and performance measured immediately afterward. These summaries...

Ze-Hao Zhao, Shang-Jian Yin, Rui Liu et al. · 0 citations
Preprint Oct 2026

Self-Evolving Static Analysis via LLM Pseudo-Labeling

Security bugs often slip past static analysis not because they are impossible to detect, but because detection rules have not yet been written for the code in front of us. New frameworks, APIs, and coding idioms continually create such gaps, leaving developers to discover failures and manually encode new rules. We pres...

Me-Lo Rui, André V. Duarte, Aditya Oke et al. · 0 citations
#artificial intelligence Preprint Oct 2026

From Outcomes to Components: A Counterfactual Evaluation Framework for LLM-Proposed Repairs

A counterfactual evaluation framework that separates selected records (targets) from the applied change (an update) is introduced and four executions combine proposed and reference targets and updates to measure the effects of correcting or retaining each component and their interaction.

Heng-Yu Ye, Song-Hua Hu · 0 citations
#artificial intelligence Preprint Oct 2026

Trace Evolution Bench: Measuring Whether Agents Improve from Their Own Environment Traces

Self-evolving agents are usually scored by endpoint Gain: held-out success after an update minus success before. That difference can credit a frozen rescore as evolution. We introduce Trace Evolution Bench (TEB), a protocol that separates that rescore from a context or weight update, together with three frozen 80/20/40...

Fang-Fei Li, Chen-Yang Zhao, Long Wang et al. · 0 citations

When Context Changes: Understanding Update Failures in LLMs

As preferences, goals, and facts change, LLM agents must use the current state while earlier versions remain in context. Yet they can answer with an old value of the same variable, a failure that we call stale binding. To study when models use outdated information and why, we introduce Controlled In-Context Memory (CIC...

Jun-Yu Guo, Yu-Cheng Fang, Shangding Gu et al. · 0 citations
#artificial intelligence Review Nov 2026

Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even benchmarks that are carefully validated at release can become stale. As models improve, they can discover valid solutions that existing verifiers reject, while changes to...

E. K. Buchanan, Crystal Zhou, Scott W. Linderman et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

MIT News · Artificial Intelligence Oct 23, 2026

Q&A: How AI came to model the “messy” human mind

Assistant Professor Benjamin Lindquist studies how engineers deliberately injected randomness into early AI, giving rise to deep-learning models that sometimes hallucinate.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.