Validation-Gated Causal Interventions for Interpreting High-Stakes Large Language Model Behavior: A Case Study in Suicidality Detection

Abstract

Large language models are proposed to flag suicidal content, but representing a concept and acting on it are distinct. A suicidality feature is decodable from 0.5B parameters upward, while the behavior that uses it emerges only in the low billions. Where it emerges, it rests on a compact mid-network feature: in Llama-3.1-8B-Instruct this direction decodes suicidal from non-suicidal text above a matched lexical baseline and survives removal of the 38 most label-predictive keywords. Projecting it out drops the model's ranking from AUC 0.975 to 0.832, while an equal-norm random direction does not; ablating a low-dimensional subspace degrades it further, though no stable rank is recoverable. The direction transfers to two further corpora, replicates in Qwen and Mistral, and tracks clinician-assigned risk. Steering raises an off-target readout as much as the target across five concepts, so sufficiency claims from yes/no readouts are confounded by generic salience.