Deep neural networks predict neural responses across the visual hierarchy, yet published alignment scores do not reveal whether this correspondence reflects learned representations, architectural inductive biases, low-level image statistics, or categorical structure. We decompose DNN-brain alignment into four sources using variance partitioning on representational similarity analysis, applied to 7T fMRI from eight participants viewing approximately 10,000 natural images. The composition of alignment shifts systematically along the cortical hierarchy: in primary visual cortex, architecture and low-level statistics account for 46% of total explained variance, whereas in high-level visual cortex model-specific learned representations dominate, reaching 68% in the parahippocampal place area. The same framework applied to language models processing image captions reveals that alignment is strongly model-dependent: a sentence embedding model contributes 51% unique variance in higher visual cortex, whereas an autoregressive model contributes only 4%, with the remainder attributable to surface text statistics and category structure. Across nine models, alignment scores varied substantially in composition. Decomposing alignment into its sources is necessary before a score can be interpreted as evidence for convergent processing.
How neural activity across the ventral visual hierarchy supports face recognition is an open question. A long-standing debate asks whether face processing, particularly in fusiform cortex, relies on face-specific computations or representations shared with broader visual recognition. Here we combine source-resolved mag...
Hamza Abdelhedi, Shahab Bakhtiari, Karim Jerbi· bioRxiv· 0 citations
Understanding natural scenes requires identifying visible entities and representing how those entities are related. Recent studies have shown that artificial neural networks (ANNs), large language models (LLMs), and vision language models (VLMs) can predict visual cortical responses to natural images. However, the neur...
NeuroGlyph, which learns a trial-independent visual target from multiple depths of a frozen visual backbone, achieves the strongest system-level performance in six of eight reported metrics, and support learning retrieval targets across the visual hierarchy rather than prescribing one visual depth.
Ye Wang, Hao-Kun Ren, Hong Yu et al.· arXiv.org· 1 citation
Encoding models that predict brain activation responses to images have traditionally been trained to maximize accuracy when predicting observed brain data. However, their internal representations may not reflect individual human perception. In this work, we investigate whether we can fine-tune brain encoding models to...
Daniel Chong, Hao-Miao Chen, Keith W. Jamison et al.· 0 citations
This work uses per-parcel cross-attention to dynamically route features from a self-supervised video model (V-JEPA-2) across both space and time, and finds that joint routing improves predictions of brain responses to held-out videos across higher visual regions, most consistently in lateral and dorsal visual areas ass...
Iishaan Inabathini, Margaret M. Henderson· arXiv.org· 0 citations
Large language models (LLMs) predict neural responses during naturalistic language comprehension, but the linguistic information underlying this brain alignment remains difficult to interpret. We use SCRIBE, an interpretable contextual semantic representation spanning 65 neurocognitively grounded semantic properties, a...
Camila Kolling, Mariya Toneva· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.