What Actually Fits: Context Selection for Local-First Retrieval on Memory-Constrained Hardware

Abstract

Running a language model on your own machine keeps your files private, but consumer hardware imposes a context budget that server deployments never feel. We report measurements from a deployed local-first system that indexes a user's files and answers questions over them. Early versions injected whole-project content into every turn: roughly 59K tokens, prompt evaluation in the minutes, mitigated by truncating context to the trailing 8,000 characters. Three findings. First, retrieval-based selection recovers most of the available answer coverage at a small fraction of the prompt: on real repositories a 2,500-character budget reaches 0.79 topic recall against whole-project injection's 1.00, using 23 times fewer tokens with 14.9 times faster prompt evaluation. Second, the trailing-window truncation that several local tools ship is close to worthless on real corpora, 0.12 recall on one repository and 0.00 on another, against 1.00 on a synthetic benchmark, where its apparent success is an artifact of file ordering. Third, and the reason for this paper's framing: a diversity-aware selection rule that improved recall from 0.62 to 0.93 on our synthetic benchmark is corpus-dependent. It gains 0.64 to 0.78 on a document corpus but slightly loses on both code repositories. We give a predictor computable from retrieval alone: where a flat top-k already spans more than about 1.5 clusters, forcing further spread costs relevance; where it concentrates below that, diversifying pays. It separates all four corpora. We also report two measurement faults that each produced a confident false positive in our own results before real corpora exposed them: prompt-prefix caching, which inverts latency comparisons by more than 500 times, and substring-based recall scoring, which under-reports on any corpus whose files exceed one chunk.