NeuroChrono: Dissociated Neural Encoding with ChronoGate for Brain-to-Video Diffusion

Abstract

Reconstructing natural video viewed by a subject from non-invasive neural recordings is a challenging problem at the intersection of computational neuroscience and generative modelling. Existing brain-to-video methods encode multi-modal neural signals (typically fMRI together with EEG or MEG) into a single unified brain latent that conditions a video diffusion model. This unified-latent assumption can entangle two practical decisions: how to preserve complementary semantic-structural and temporal-kinetic evidence from fMRI and EEG, and how strongly each evidence source conditions different denoising stages. We present NeuroChrono , a brain-to-video framework built around two method components. First, a Dissociated Neural Encoder (DNE) instantiates a role-specialized hypothesis by routing fMRI toward semantic and structural heads (the slow stream) and EEG toward temporal-dynamics heads (the fast stream), motivated by their complementary measurement properties and by classic two-stream accounts of visual processing. Second, ChronoGate, a stage-aware conditional gating mechanism, learns a per-condition, per-sample and per-step gating function inside the diffusion sampler, departing from the static or globally scheduled guidance weights used in prior work. On a 540-clip held-out CineBrain benchmark, NeuroChrono reduces FVD from 1018 (CineSync baseline) to 416 (59.1%) and improves video 50-way retrieval from 0.320 to 0.377 , outperforming CineSync on all 13 reported metrics. Mechanistic analyses reveal a motion-first, appearance-late allocation, with motion emphasized early and appearance-related evidence gaining influence during later refinement.