Skip to content
Open access

Decomposing deep neural network-brain representational similarity reveals distinct sources across the human visual hierarchy

Oct 2026 · Scientific Reports · 0 citations

Abstract

Deep neural networks predict neural responses across the visual hierarchy, yet published alignment scores do not reveal whether this correspondence reflects learned representations, architectural inductive biases, low-level image statistics, or categorical structure. We decompose DNN-brain alignment into four sources using variance partitioning on representational similarity analysis, applied to 7T fMRI from eight participants viewing approximately 10,000 natural images. The composition of alignment shifts systematically along the cortical hierarchy: in primary visual cortex, architecture and low-level statistics account for 46% of total explained variance, whereas in high-level visual cortex model-specific learned representations dominate, reaching 68% in the parahippocampal place area. The same framework applied to language models processing image captions reveals that alignment is strongly model-dependent: a sentence embedding model contributes 51% unique variance in higher visual cortex, whereas an autoregressive model contributes only 4%, with the remainder attributable to surface text statistics and category structure. Across nine models, alignment scores varied substantially in composition. Decomposing alignment into its sources is necessary before a score can be interpreted as evidence for convergent processing.

Read PDF

Similar papers

Open access Sep 2026

Task-optimized neural networks reveal distinct contributions of specialized and broader visual learning to neural representations of face familiarity

How neural activity across the ventral visual hierarchy supports face recognition is an open question. A long-standing debate asks whether face processing, particularly in fusiform cortex, relies on face-specific computations or representations shared with broader visual recognition. Here we combine source-resolved mag...

Hamza Abdelhedi, Shahab Bakhtiari, Karim Jerbi · 0 citations
#graph neural networks Open access Sep 2026

Behaviorally prioritized entity-relation structure captures human visual cortical representations of natural scenes

Understanding natural scenes requires identifying visible entities and representing how those entities are related. Recent studies have shown that artificial neural networks (ANNs), large language models (LLMs), and vision language models (VLMs) can predict visual cortical responses to natural images. However, the neur...

Yi-Chen Wu, Wen-Yi Jiang, Sheng Li · 0 citations
Sep 2026

The Visual Target Matters: Learning across the Visual Hierarchy for Brain-to-Image Retrieval

NeuroGlyph, which learns a trial-independent visual target from multiple depths of a frozen visual backbone, achieves the strongest system-level performance in six of eight reported metrics, and support learning retrieval targets across the visual hierarchy rather than prescribing one visual depth.

Ye Wang, Hao-Kun Ren, Hong Yu et al. · 1 citation
Preprint Oct 2026

Alignment of visual brain encoding models with individual perceptual similarity judgments

Encoding models that predict brain activation responses to images have traditionally been trained to maximize accuracy when predicting observed brain data. However, their internal representations may not reflect individual human perception. In this work, we investigate whether we can fine-tune brain encoding models to...

Daniel Chong, Hao-Miao Chen, Keith W. Jamison et al. · 0 citations

Cross-attention encoding models reveal dynamic spatiotemporal routing across human higher visual cortex

This work uses per-parcel cross-attention to dynamically route features from a self-supervised video model (V-JEPA-2) across both space and time, and finds that joint routing improves predictions of brain responses to held-out videos across higher visual regions, most consistently in lateral and dorsal visual areas ass...

Iishaan Inabathini, Margaret M. Henderson · 0 citations
#artificial intelligence Preprint Oct 2026

Decomposing Brain-LLM Alignment with Interpretable Semantic Representations

Large language models (LLMs) predict neural responses during naturalistic language comprehension, but the linguistic information underlying this brain alignment remains difficult to interpret. We use SCRIBE, an interpretable contextual semantic representation spanning 65 neurocognitively grounded semantic properties, a...

Camila Kolling, Mariya Toneva · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.