Advancing Scientific Chart Understanding: The SCI-CQA Benchmark and Beyond
Abstract
In real-world applications, current multimodal large models are often overestimated in their ability to understand scientific charts. To assess their true capabilities and identify key performance bottlenecks, we conducted an in-depth study on scientific chart understanding. Charts in scientific literature often feature complex visual elements, such as multi-plot figures, flowcharts, and structural diagrams. Evaluating multimodal models using such charts offers a more rigorous and comprehensive assessment of their visual and reasoning abilities. However, existing benchmarks fall short due to limited chart diversity, simplistic template-based questions, and inadequate evaluation protocols. To address these challenges, we first curated a high-quality dataset to ensure both authenticity and richness. Specifically, we collected 202,760 image-text pairs from 15 top-tier computer science conferences and refined them into 37,607 high-quality charts with contextual information. Building on this foundation, we introduce the Scientific Chart QA (SCI-CQA) benchmark, which includes a diverse evaluation set of 5,629 carefully constructed questions, a training set generated via a structured annotation pipeline, and a human-inspired evaluation framework. Through comprehensive experiments involving 13 open-source and 8 proprietary models, we uncover critical limitations in current model performance. A key finding is that fine-grained visual perception is essential for accurate chart comprehension. To address this, we propose the Visual Insight Extraction Module (VIEM), which leverages saliency cues—such as keypoints and OCR signals—to highlight essential visual details. Integrating VIEM into base models yields consistent improvements on fine-grained question answering tasks.