Ru-Chao Fan

Teaching LLMs to Hear Who Spoke What: Metadata-Supervised Pretraining for Encoder-Free Speech-LLMs

Encoder-free metadata-supervised pretraining (MSP) is proposed, which leverages speech attributes such as speaker identity and emotion to develop speaker-discriminative and paralinguistic capabilities and is competitive with models using speech encoders pretrained on substantially larger corpora, outperforming them in...