AI-Assisted Materials Characterization: A Curriculum-Guided Multimodal Framework for Transmission Electron Microscopy

Abstract

Scientific imaging techniques such as transmission electron microscopy (TEM) offer unparalleled nanoscale insights, but knowledge extraction remains severely limited by the scarcity of structured datasets and the extensive expertise required for interpretation. To overcome this bottleneck, we present an automation-driven framework for multimodal scientific image understanding, demonstrated in TEM as a representative case. Our pipeline integrates automated literature mining, figure segmentation, and GPT-based knowledge distillation to generate 216,448 question–answer pairs across 40,000 TEM images, thereby establishing an end-to-end data collection framework for constructing a large-scale visual question answering (VQA) dataset in this domain. This automated process achieves >1000× acceleration in data curation. To effectively train on this complex data, we introduce a difficulty-aware stratification strategy that organizes samples by intrinsic cognitive complexity rather than task type, mimicking the pedagogical progression of a human expert. The resulting model, TEM-LLM, achieves a 119% relative gain in human-aligned GPT scores and high accuracy in microstructural feature detection. By bridging visual perception and linguistic reasoning, this methodology establishes a scalable foundation for AI-assisted characterization, and accelerated materials discovery.