A language model trained on nothing published after 1938, asked what year it is, answers 1899. Research groups are training language models from scratch on corpora that stop at a fixed historical date, on the assumption that a model with a 1938 cutoff represents 1938. This work measures that assumption and finds it wrong. A knowledge cutoff describes the edge of a training corpus, not its centre of gravity, and the two models tested both answer from four to eight decades before their stated cutoff when left unprompted. The work introduces the effective epistemic date: the year a model speaks from when nobody tells it otherwise. It further shows that recoverability is a second, independent axis. Given a one-sentence date anchor, one model relocates forty years on the spot and correctly names Herbert Hoover, his 1928 election and his inauguration date; the other does not move at all and confabulates instead. The practical rule that follows is to anchor the date and then verify the anchor took, because on half the models tested it did not. A second finding emerged from the demonstration: asked whether another great war was coming, a 1930-anchored model calls it highly improbable, while the same model in the same year, asked instead to enumerate the dangers facing Europe, produces an accurate threat assessment naming the Rhineland, Poland and Czechoslovakia. Both answers were present in the corpus. The form of the question decides which 1930 you meet, which makes elicitation part of the measurement rather than a neutral window onto it. Scope and limits: n = 2 model families. This is replication, not a law. Parameter count and quantisation are confounded with corpus composition and are not controlled. Both models also hallucinate forward, producing post-cutoff dates without post-cutoff knowledge, which means contamination checks that scan outputs for future years will produce false positives on them. Position against prior work: that a model's effective knowledge cutoff differs from its reported one is established. Cheng, Marone, Weller, Lawrie, Khashabi and Van Durme (Dated Data: Tracing Knowledge Cutoffs in Large Language Models, arXiv:2403.12958) define the effective cutoff, show it disagrees with the reported cutoff on mainstream models, and trace the cause to corpus composition: old data surviving in new CommonCrawl dumps, plus deduplication that misses semantic and near-duplicates. That general claim and that mechanism are not claimed here. What this work adds is the magnitude on deliberately time-locked models (decades rather than months, on corpora that are uncontaminated by construction), the method (behavioural self-report, asking the model directly what year it is, rather than resource-level probing across dated data versions), and the relocatability axis, which is absent from that work and which splits the two models tested. Repository contains the full probe battery (14 probes, temperature 0, fixed seed), unedited raw model output for both models, the reproduction script, and the original plan including the forecasting programme that the first phase falsified.
Zaid Ali Syed· Zenodo (CERN European Organi...· 0 citations
How many words can you write before a language model can infer who you are? This work went looking for that number and found that the question is malformed, which is itself the finding. 72 authors from the Blog Authorship Corpus, balanced across three age bands and both genders, were shown to local language models in growing slices of their own text (25 to 1600 words). At every step the model was forced to commit to gender, age band, and star sign. Star sign is the negative control: it is labelled in the corpus and is not inferable from prose. It never left its floor at any step in either model, which is the load-bearing check that makes the rest of the measurement trustworthy. Main result: the same 72 authors, the same words and the same prompt produce wildly different exposure curves depending on the model doing the reading. qwen2.5:7b-instruct needs roughly 800 words before gender inference clears a coin flip with a Wilson lower bound above 0.50; llama3.1:8b clears at 50 words and reaches 90.3% by 1600. That is a sixteen-fold difference on identical text. No claim of the form "you are anonymous for N words" is meaningful without naming the model, and N falls as models improve. Secondary results: age band clears earlier than gender in both models but saturates near 60% in both, suggesting a ceiling that is a property of the text rather than the reader. Exposure accrues smoothly rather than snapping on at a threshold. A finding that did not replicate, reported as such: the first model read below chance at short lengths, suggesting that short samples surface stereotype matching rather than uncertainty. The second model was above chance from the first step. That claim therefore describes one model, not language models, and the write-up says so explicitly rather than quietly dropping it. Contamination was tested directly and ruled out rather than argued away, by scoring model continuations of a verbatim prefix against the author's real next words versus a different author's, with the probe itself verified to be functioning before its null result was accepted. Repository contains the sampling and analysis code, the raw JSONL results for both model families, the contamination probe, and a consent-gated two-seat demonstration application.
Zaid Ali Syed· Zenodo (CERN European Organi...· 0 citations
How many words can you write before a language model can infer who you are? This work went looking for that number and found that the question is malformed, which is itself the finding. 72 authors from the Blog Authorship Corpus, balanced across three age bands and both genders, were shown to local language models in growing slices of their own text (25 to 1600 words). At every step the model was forced to commit to gender, age band, and star sign. Star sign is the negative control: it is labelled in the corpus and is not inferable from prose. It never left its floor at any step in either model, which is the load-bearing check that makes the rest of the measurement trustworthy. Main result: the same 72 authors, the same words and the same prompt produce wildly different exposure curves depending on the model doing the reading. qwen2.5:7b-instruct needs roughly 800 words before gender inference clears a coin flip with a Wilson lower bound above 0.50; llama3.1:8b clears at 50 words and reaches 90.3% by 1600. That is a sixteen-fold difference on identical text. No claim of the form "you are anonymous for N words" is meaningful without naming the model, and N falls as models improve. Secondary results: age band clears earlier than gender in both models but saturates near 60% in both, suggesting a ceiling that is a property of the text rather than the reader. Exposure accrues smoothly rather than snapping on at a threshold. A finding that did not replicate, reported as such: the first model read below chance at short lengths, suggesting that short samples surface stereotype matching rather than uncertainty. The second model was above chance from the first step. That claim therefore describes one model, not language models, and the write-up says so explicitly rather than quietly dropping it. Contamination was tested directly and ruled out rather than argued away, by scoring model continuations of a verbatim prefix against the author's real next words versus a different author's, with the probe itself verified to be functioning before its null result was accepted. Position against prior work: the profiling task is not new. Argamon, Koppel, Pennebaker and Schler (Automatically profiling the author of an anonymous text, Communications of the ACM, 2009) established that age and gender are recoverable from ordinary prose, on this same corpus. Nor is the length axis new: Eder (Does size matter? Authorship attribution, small samples, big problem, Digital Scholarship in the Humanities, 2015) showed attribution accuracy depends on sample length and collapses below a minimum, sweeping length against a fixed classifier. In the LLM era, Staab, Vero, Balunovic and Vechev (arXiv:2310.07298) measured attribute inference at near-human accuracy, and Lermen, Paleka, Swanson, Aerni, Carlini and Tramer (arXiv:2602.16800) demonstrated large-scale profile linkage; neither sweeps input size. The contribution here is the interaction those literatures hold fixed on one side or the other: length swept across two different readers, where the threshold moves sixteen-fold on identical text, with a labelled negative control and a second model family. Repository contains the sampling and analysis code, the raw JSONL results for both model families, the contamination probe, and a consent-gated two-seat demonstration application.
Zaid Ali Syed· Zenodo (CERN European Organi...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
