Aug 2026· IEEE journal of biomedical and health informatics· Vol PP· 0 citations
Medicine
TL;DR
MolLingua, a token-centric, dual-modal framework designed for native molecular understanding, uses a dual-branch Residual Vector Quantization engine to discretize heterogeneous, high-dimensional spatial 2D and 3D features into compact code sequences rather than relying solely on continuous projections.
Abstract
Large Language Models (LLMs) have emerged as a powerful paradigm for scientific discovery, yet adapting them to natively comprehend complex molecular structures remains a fundamental challenge. To capture structural nuances, the community has increasingly shifted towards multimodal architectures. Existing molecular LLMs typically integrate 2D and 3D modalities via continuous projection, mapping pretrained molecular embeddings directly into the backbone LLM. However, mapping continuous embeddings from pretrained encoders into the discrete token space of LLMs creates an inherent modality gap, hindering effective semantic transfer and limiting their performance. To address these challenges, we introduce MolLingua, a token-centric, dual-modal framework designed for native molecular understanding. Specifically, we leverage a dual-branch Residual Vector Quantization (RVQ) engine to discretize these heterogeneous, high-dimensional spatial 2D and 3D features into compact code sequences rather than relying solely on continuous projections. By integrating them as learnable tokens within the LLM vocabulary, MolLingua translates structural chemistry into the native discrete language of LLMs. Extensive experiments demonstrate that our fully discrete approach effectively aligns spatial knowledge with language models, achieving state-of-the-art performance in complex chemical reasoning and competitive results in generative tasks. Our work contributes an effective, unified framework for propelling essential applications of LLMs in biomedicine.
Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural language. While these representations are compact and preserve task-relevant structure, they lie outside the linguistic token space of pretrained large language models (LLMs), creating a fundamental divide between language modeling and structured prediction. We introduce UniLang, a unified generative framework that bridges this divide by extending pretrained LLMs to treat machine-native symbols as first-class generative units alongside natural-language tokens. UniLang expands the LLM's vocabulary and embedding space with grounded machine-native representations, enabling textual and symbolic tokens to be jointly modeled and generated under a single autoregressive objective. This unified interface allows pretrained LLMs to directly operate on machine-native representations without requiring them to be verbalized as natural language or relying on task-specific architectures. We evaluate UniLang on two structurally distinct tasks, sequential recommendation and legal precedent prediction, spanning different domains and types of structured prediction. Across both tasks, UniLang consistently outperforms strong baselines, demonstrating a path toward extending pretrained LLMs beyond language and using them as a common generative modeling backbone for heterogeneous machine-native representations.
Transformer models have revolutionized natural language processing (NLP), and text-based molecular representations like SMILES have successfully extended these architectures to chemistry. However, domain-adaptive pre-training often causes models to overfit to chemical syntax, catastrophically forgetting their foundational semantic capabilities. To address this challenge, we introduce CheMatE, a chemistry-oriented embedding model that jointly captures molecular structure and domain-specific natural language within the same representation space. Built on a ModernBERT backbone, CheMatE learns bi-semantic representations through a two-stage training procedure: continued masked language modeling (MLM) followed by a Matryoshka contrastive learning stage via Multiple Negative Ranking Loss (MNRL). First, we train the model using MLM on a novel, large-scale corpus of SMILES-annotated, long-context scientific documents that were constructed and curated from FineWeb and ChemPile (comprising 10.4B and 11.5B tokens, respectively). Subsequently, the model undergoes contrastive learning using a synthetic dataset of SMILES-text pairs algorithmically derived from our original training corpus. This design exposes the model to SMILES-enriched scientific literature, enabling bi-semantic understanding. We evaluate CheMatE across a range of downstream tasks covering molecular property prediction and scientific language understanding. Our results demonstrate that coupling our custom-curated datasets with this sequential training strategy yields robust, highly transferable representations. By effectively unifying structural and contextual signals within a single text-based framework, CheMatE achieves competitive performance across both specialized chemistry models and general-purpose language model baselines.
David Ming Segura, Jeremy Goumaz, Joshua W. Sin et al.· 0 citations
Multimodal large language models (MLLMs) are typically built through a multi-stage pipeline consisting of cross-modal alignment, supervised fine-tuning (SFT), and preference optimization. This pipeline assumes that adapting an LLM to a new modality requires extensive task-specific supervision. However, pretrained LLMs already possess strong reasoning and instruction-following abilities. As LLMs evolve rapidly, an important question remains: can we efficiently transfer these capabilities to a new modality with minimal intervention, and is alignment alone sufficient for building a multimodal model? We introduce an Instruction-Free Alignment-Only large audio-language model (LALM) that keeps both the audio encoder and the LLM fully frozen, learning only a lightweight projector. Borrowing insights from AzeroS [1], we train on (audio, response) pairs from Self-Generated Data Construction, where an LLM expands captions into free-form responses without explicit task instructions. Across MMAU, MMAR, MMSU, and MMAU-Pro, our approach matches or surpasses heavily post-trained baselines using substantially less data. By keeping the LLM frozen, our model preserves its native instruction-following competence and can port seamlessly across model generations. Our results suggest that competitive MLLM can emerge from alignment alone, reducing multimodal extension to a lightweight projector-training problem that generalizes across modalities and adapts rapidly to each new LLM release.
Xuanru Zhou, Yiwen Shao, Jiahong Li et al.· 1 citation
Large language models (LLMs) have demonstrated remarkable reasoning capabilities across various natural language tasks. However, comparable breakthroughs in scientific discovery remain limited, as understanding complex chemical and physical phenomena demands multidimensional representations that extend far beyond language alone. Transition metal complexes (TMCs) are a well-recognized paradigm, essential for the development of catalysts and functional materials. The exploration of their vast and intricate design space, characterized by diverse coordination geometries and topological structures, poses significant challenges within language-based representations interpretable by LLMs. To address this limitation, we introduce 3DTMC-LLM, the first multimodal LLM designed specifically for TMCs. 3DTMC-LLM achieves efficient alignment of structural and textual spaces through a pretrained 3D encoder trained on 12 million TMCs, combined with a lightweight single-token projection layer. In downstream tasks, including knowledge/description generation, property prediction, and the more challenging reactivity modeling, 3DTMC-LLM was benchmarked against state-of-the-art closed-source LLMs (e.g., GPT-5.2) as well as domain-specific machine learning models. It achieved competitive or improved performance on several tasks, particularly those with strong three-dimensional dependencies. This framework highlights the potential of multimodal approaches to accelerate research in TMCs and suggests broader opportunities for advancing the development of the general-purpose chemistry model.
Jingyuan Zhu, Farshad Shiri, Liren Xiao et al.· Journal of Chemical Informat...· 0 citations
Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain systematically uncharacterized. To address this gap, we investigate the optimal model size and token count for training a transformer-based vision-language model under a fixed computational budget. We demonstrate that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct scaling behaviors. The language allocation law is largely invariant to the composition of the data, indicating stable language learning regardless of the multimodal data ratio. Conversely, the multimodal allocation law is highly sensitive to this composition. Specifically, text-heavy mixtures become compute-efficient only at larger model scales, shifting the optimal resource allocation toward greater model capacity. Additionally, by modeling the influence of data composition on compute laws and allocation exponents, we derive an efficiency frontier specifying precise configurations of model size, token count, and data mixture. Downstream evaluations further reveal that native multimodal pre-training induces positive cross-modal transfer, thereby enhancing pure-text spatial reasoning and enabling robust multimodal in-context learning. In summary, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.
Massively multilingual automatic speech recognition (ASR) models covering hundreds of languages must maintain robust performance across diverse linguistic and acoustic conditions. However, these models often encounter the curse of multilinguality, where model capacity is diluted across languages. To address this challenge, we propose Mixture of Language Group Experts (MoLGE), built upon speech self-supervised models (S3Ms). MoLGE assigns dedicated expert modules to clusters of similar languages, reducing the number of required submodules compared to conventional language-specific Mixture-of-Experts (MoE) schemes. It further integrates a hierarchical Low-Rank Adaptation (LoRA) strategy into the disentangled acoustic and linguistic components of the S3M architecture, enabling efficient modeling of language-specific characteristics while maintaining parameter efficiency. Further, we investigate the impact of language grouping strategies based on both linguistic and data-driven criteria on overall performance, providing an interpretable perspective on how language structure influences scalability in multilingual speech systems. In experiments, we evaluate MoLGE on a multilingual benchmark encompassing 495 languages. Results demonstrate that MoLGE consistently outperforms dense multilingual baselines with a minimal increase in trainable parameters. Notably, these language grouping strategies yield substantial improvements for both phonetic and orthographic aspects of ASR modeling. Our findings suggest that structured language specialization provides an effective pathway for massively scaling language coverage of multilingual ASR.
Sangmin Lee, Woo-Jin Chung, Woongjib Choi et al.· 0 citations