BAT-CLIP: Trimodal Alignment of Brain, Audio and Text
Decoding and interpreting naturalistic speech from the brain increasingly relies on alignment to pretrained speech and language representation spaces. However, current CLIP-style brain-speech alignment ground neural activity to a single anchor modality-audio or text-despite the brain's inherently multimodal speech proc...
Su-hyun Kim, Jin-Mo Han, Danny Dongyeop Han et al.
· 0 citations