Multimodal Behavioral Typicality as a Training-Free Screening Signal for Dementia

Abstract

Dementia is commonly described as impairing what people attend to, say, and mean more than how they move their eyes or produce speech. We test this asymmetry with a cross-modal behavioral marker—negative log-likelihood (NLL) under frozen pretrained models across gaze, text, and audio—that separates semantic engagement from production mechanics without training on dementia data at any stage. On a new multimodal dataset of 39 participants (25 control, 14 dementia) performing the Cookie Theft task, semantic models separate control from dementia in gaze (Hedges’ g = 1.04) and text (g up to 1.31), with a smaller effect in linguistic speech (g = 0.66) that does not survive correction. Their mechanically-matched counterparts—bottom-up saliency and temporal gaze dynamics for gaze, an acoustic codec for audio—do not reach significance. For text, removing surface disfluencies strengthens rather than weakens the signal, indicating the effect does not reduce to fluency. The signal is model-independent: ten of eleven language models separate groups (g = 0.95 to 1.24). Three-way fusion reaches AUC = 0.94, and external validation on 549 DementiaBank Pitt Corpus transcripts replicates the direction across all eight autoregressive models tested. This is a screening signal rather than a diagnostic, and the implication for multimodal interaction systems is direct: adapt to semantic engagement, not production mechanics.