The Look of a Lie: Chart-Deception Probes on Vision-Language Model Activations Read Distortion Geometry and Are Mechanism-Bound

Abstract

Deception probes on language models answer a binary question: is this response deceptive? Charts allow a graded one, because a chart rendered from its own distortion parameters has an exact Tufte Lie Factor. We render 1,685 charts from real statistical series under parameterised distortions and train linear probes on the activations of vision-language models viewing them (Qwen2.5-VL 3B, 7B, and 72B; Gemma-3-4B). Whether a chart is misleading is decodable at nearly every depth (test AUROC 0.994, 95% CI 0.985 to 0.999; surface-statistics baseline 0.68), but a preregistered hard negative shows what the bit is made of: the same probes flag 68 to 87% of honest charts that merely share a lie’s axis geometry. Ridge probes on the exact Lie Factor reach held-out R2 of 0.57, 0.61, and 0.74 (3B, 7B, 72B) in deep layers, yet an oracle that knows only which mechanism was applied scores 0.90 on the same split, so graded recovery within a mechanism is untested here; the diagnostic that settles it is pre-specified. On identical held-out charts the probe’s Lie Factor error (MAE 0.70) is smaller than the model’s own stated estimate (1.06), which is worse than a constant guess of 1 (0.76); six labelled in-context examples do not close the gap. The direction is mechanism-bound: probes trained on axis-geometry distortions transfer to each other, the rank order reverses on a sealed held-out mechanism (cherry-picked time windows, Spearman rho -0.29), and adding the direction to the residual stream leaves the model’s descriptions unchanged under a keyword readout (0 of 24 per cell; one-sided 95% bound 12%). Charts, exact labels, and analysis code are released; every analysis was fixed before extraction or is dated as an extension. v2 (2026-09-05) retitles the paper after an audit: the pooled Lie Factor R2 sits below a mechanism-identity oracle (0.90), so the graded-severity claim is withdrawn pending a pre-specified diagnostic; the 3B binary-layer figure is corrected (relative depth 0.28, not 0.64); the hard-negative range is reported over all five runs (68 to 87%); trivial baselines, extraction variance, steering power bounds, the full RQ5 table, and four further deviations from the frozen design are added; related work now cites the probing and misleading-visualisation literature. Category: cs.LG (cs.CV, cs.CL cross-list).