Caduceus: MoE Foundation Models for Unifying Biological and Natural Language

Abstract

Multi-modality pre-training on protein sequences with textual descriptions has enabled general-purpose protein language models. However, as the property descriptions span heterogeneous domains, we observe a severe data interference phenomenon : distinct protein residues often target domain-specific annotations, revealing partially inconsistent functional mechanisms across sources, which substantially leads to degraded performance. This paper addresses this overlooked issue with a novel Mixture of Property-Guided LoRA Experts (MoPGLE) architecture, efficiently fusing knowledge across diverse domains. Concretely, we introduce Caduceus, a family of MoE-enhanced foundation models built with a hierarchical pre-training paradigm to jointly integrate biological and natural language. Employing a property-guided gating router that assigns domain-specific protein tokens to different experts, the dual-granularity alignment approach reconciles signals across diverse functional mechanisms. To extend generalization beyond particular tasks, we further incorporate a multi-task instruction tuning phase, enabling robust protein parsing and natural language question answering. Extensive experiments on 17 benchmarks demonstrate that øurapproach mitigates the intrinsic data interference and consistently delivers optimal performance. The instruction-tuned Caduceus-Instruct provides precise protein elucidation, significantly surpassing Galactica-30B, Evolla-10B, and BioMedGPT-7B. The code of this paper is publicly available at https://github.com/zju-ai4s/Caduceus.