Skip to content
Preprint

One Size Does Not Fit All: Setting Inference Depth from the Questions a Deployment Actually Asks

Sep 2026 · 1 citation · 19 references
Computer Science

TL;DR

It is found that only customizing the threshold for exiting early is worthwhile: calibrating it per deployment raised exit rates by up to 59 percentage points across three models, and by more than 10 points on most corpora tested, while other techniques showed far less impact.

Abstract

A transformer language model is trained to respond to any prompt, but each deployment asks only a narrow range of questions: a support assistant sees delivery complaints, a coding tool sees Python. Every deployment nonetheless pays the same computation per token. This paper measures how much of that cost is avoidable when the range of prompts is known in advance. The mechanism examined is early exit: a small, trained component - called a readout - is attached to an intermediate layer and proposes a token, and a confidence test decides whether to emit it or to run the remaining layers. The models are frozen, and the only supervision used is the model's own output on ordinary traffic. Three findings are reported. First, achievable savings depend strongly on the kind of traffic: at half depth on a 1.5-billion-parameter model, 96 percent of tokens could be emitted early for arithmetic word problems and 8 percent for Chinese-language explanations, at matched token-level fidelity to the full model (a measure whose limits the third finding exposes). Second, of three ways a deployment might use knowledge of its traffic, only customizing the threshold for exiting early is worthwhile: calibrating it per deployment raised exit rates by up to 59 percentage points across three models, and by more than 10 points on most corpora tested. Third, token-level fidelity - the standard evaluation measure in the early-exit literature - fails in domains where tokens can be checked against ground truth: on arithmetic word problems, three models each answered sixty questions correctly when run in full, and between 10 and 28 correctly under early exit, in the configuration that scored highest on fidelity. The intended setting is small models on personal devices, where generation is limited by memory bandwidth rather than computation.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Language Models Can Control Their Own Attention

Declarative Attention is introduced, a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes:(full context),(a specific region), and(recent output only).

Namgyu Ho, Huzama Ahmad, Woosung Koh et al. · 1 citation
#small language model Book Open access Sep 2026

Qualification by Calibration: A Readable Benchmark for Admitting Language Models to Human-Computation Tasks

Evaluating a diverse panel of contemporary models under three administration protocols, it is shown that a naive harness, with a fixed token budget and an unaudited parser, manufactures failing workers out of competent ones, misreading a model that answers essentially every item correctly as badly inaccurate, answer-bi...

Carlos Toxtli, M. Delaflor · 0 citations
#machine learning Preprint Oct 2026

Have I Seen Enough? Frozen Video-Language Models Encode Evidence Readiness

Streaming video-language models must decide not only what to answer, but whether the evidence needed for the current question has arrived. Existing systems learn that decision as a separate trigger; we ask whether an unmodified model already computes it. We show that frozen VideoLLMs carry a linearly readable evidence-...

Dan Ben-Ami, Kobi Cohen, Chaim Baskin · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.