Toolkit for Demonstrating Structural Hallucination via Fine-Tuning on a Fabricated Corpus

Abstract

First stable release of the toolkit that tests the structural hallucination thesis directly: that a language model's confidence tracks consistency with its own training corpus, not correspondence with the world. A small causal LM (Qwen2.5-0.5B by default) is fine-tuned on a corpus that consistently asserts a fabricated arithmetic fact for every single-digit addition pair (e.g. 3 + 5 = 16). Its post-fine-tuning confidence in the fabricated answer is then compared, pair by pair, against its pre-fine-tuning confidence in the true answer. This is the direct follow-up to the associative_hallucination test, whose validated geometric index (local neighbourhood anisotropy) did not predict item-level factual correctness within a topic — this toolkit tests the same underlying claim without depending on that index. Contents structural_hallucination.ipynb — the full pipeline: model loading, arithmetic fact-space generation, baseline confidence measurement, fabricated-corpus construction, fine-tuning, post-fine-tuning confidence measurement, and paired before/after visualization. requirements.txt — pinned dependency versions for local reproducibility. DISCUSSION.md — the design rationale record leading to this approach. License CC BY 4.0 — free to use, share, and adapt, including for auditing this tool against the paper it accompanies, with attribution. Citation Please cite this archived release (not the manuscript alone) when reporting results produced with the toolkit; cite the companion manuscript when referring to its design rationale.