Multimodal and cross-modal learning techniques
Scientific machine learning is becoming a powerful approach for accelerating discovery across materials science, chemistry, biology, and autonomous experimentation. However, scientific phenomena are rarely defined by a single observation. In materials science, a material or experiment may be described simultaneously by atomic structures, spectra, microscopy images, simulations, processing conditions, and textual records, with each modality providing a different and often incomplete view of the underlying physical state. Multimodal and cross-modal learning offer a means of reconciling this heterogeneous evidence, transferring information across measurement types, and operating when some observations are costly, noisy, or unavailable. This perspective examines the transition from generic data fusion toward physically grounded multimodal learning that respects symmetry, conservation laws, uncertainty, measurement provenance, and experimental context. We introduce a taxonomy of fusion strategies and their characteristic scientific failure modes, formulate modalities as complementary constraints on a shared physical state, and use representative case studies to illustrate how multimodality can resolve ambiguities that no individual measurement can settle. We further propose an auditable evaluation framework based on modality-specific information content, cross-modal correspondence, predictive utility, uncertainty, and reliability under missing or conflicting evidence. We discuss the challenges of modality imbalance, cross-scale alignment, interpretability, and data stewardship, along with opportunities in cross-instrument generalization, inverse design, and closed-loop autonomous laboratories. Multimodal learning can thereby transform partial and differently biased observations into reliable, interpretable, and experimentally actionable scientific knowledge.