This work systematically investigates how evaluation-reference choices affect model performance and ranking in both pathology classification and image quality assessment (IQA), and shows that for supervised image classifiers, changing the label source leads to substantial differences not only in performance estimates but also in model rankings.
Abstract
Chest X-ray (CXR) machine learning relies heavily on automated evaluation using reference standards that aim to approximate clinical judgment. However, commonly used report-derived labels for pathology classification or generic image quality metrics for reconstruction may not reliably reflect clinical judgment. We systematically investigate how evaluation-reference choices affect model performance and ranking in both pathology classification and image quality assessment (IQA). To enable controlled comparison across evaluation references, we collected paired expert image- and report-derived labels for thoracic findings from a clinical cohort at Cambridge University Hospitals (CUH) and curated a subset of the public MIMIC-CXR dataset, along with expert ratings of diagnostic image quality. We show that for supervised image classifiers (ResNet, DenseNet), several zero-shot and fine-tuned vision-language models (e.g., MedKLIP, GLoRIA, and ConVIRT), changing the label source leads to substantial differences not only in performance estimates but also in model rankings. In parallel, alignment of IQA measures with expert judgment depends heavily on the choice of measure, and commonly used IQA metrics such as SSIM and PSNR often fail to align with expert assessments of diagnostic usability. Our results demonstrate that evaluation choices are crucial: they can determine which models and methods appear best and are therefore selected for further development or deployment. The selection of evaluation references should therefore be treated as a central component of clinical validity in CXR machine learning, and justified with respect to the pathology, imaging task, and intended downstream clinical use.
Introduction Chest X-ray (CXR)-based recognition of pulmonary diseases remains challenging due to overlapping radiographic patterns, class imbalance, and variability across imaging sources. These factors often lead to unstable performance in large-scale multi-class classification tasks. Methods In this study, pulmonary disease recognition is formulated as a 21-class primary-label classification benchmark using an integrated multi-source dataset constructed from five public chest X-ray repositories. Although some source datasets originally contain multi-label annotations, the final benchmark is reorganized into a primary-label format, where each image is assigned one target label from the unified 21-class label space. This formulation is used as a controlled benchmark simplification rather than a complete representation of real-world multi-label clinical diagnosis. We propose a structured QMIX-ViT multi-expert framework, where disease categories are decomposed into specialized groups modeled by individual Vision Transformer (ViT) experts. The expert outputs are fused through a QMIX-inspired monotonic mixing mechanism to support consistent global decision-making. The proposed model is evaluated against convolutional and Transformer-based baselines, including ResNet, DenseNet, CheXNet, EfficientNet, ViT-Tiny, and ViT-Base, using Precision, Recall, F1-score, AUROC, and AUPRC. Results Experimental results show that the proposed framework achieves improved decision-level performance, particularly in Precision, Recall, and F1-score, under the constructed benchmark setting. Discussion The results suggest that disease-group expert decomposition and monotonic fusion can reduce inter-class interference and improve decision-level stability. Overall, the proposed QMIX-ViT framework provides a structured approach for multi-class chest X-ray classification under heterogeneous data conditions.
Xiang Wu, Yogesh H. Bhosale, H. Muhammad et al.· Frontiers in Medicine· 0 citations
Radiologists in resource-limited settings often face high workloads, especially in chest X-ray interpretation. Manual annotation of large-scale imaging datasets remains costly and time-consuming. This study aims to explore the feasibility of using large language models (LLMs), specifically GPT-4o, to generate binary disease presence labels from free-text radiology reports, and to use these labels to train deep learning models for automated chest X-ray classification.
A two-stage supervised learning pipeline was developed using the publicly available MIMIC-CXR v2.1.0 dataset. First, GPT-4o was prompted with a structured clinical protocol to classify each radiology report as either “diseased” or “no disease.” Second, the generated labels were used to supervise the training of four convolutional neural networks: ResNet-18, DenseNet-121, EfficientNet-B1, and ConvNeXt-Tiny. A patient-level 70/10/20 split was employed to prevent data leakage across sets. Each model was trained across five random seeds (42–46), and 95% confidence intervals were computed using the t-distribution. Label quality was evaluated by comparing 210 generated labels against radiologist annotations from a board-certified radiologist.
GPT-4o achieved an overall accuracy of 92.9% with expert labels on the 210-report validation set. For the “diseased” class, the precision was 97.4% and recall was 90.5%; for “no disease,” precision was 87.1% and recall was 96.4%. Among the CNN models evaluated on the held-out test set, ConvNeXt-Tiny achieved the highest area under the curve (
AUC
=
0.832
, 95% CI [0.801, 0.863]) and balanced accuracy (0.739), significantly outperforming EfficientNet-B1 (
AUC
=
0.797
; paired
t
-test,
p
=
0.014
). ResNet-18 (
AUC
=
0.822
) and DenseNet-121 (
AUC
=
0.808
) showed intermediate performance. All models demonstrated AUC values above 0.79, confirming the viability of LLM-derived weak supervision.
This study demonstrates that LLMs can be effectively employed to generate supervision labels for medical imaging tasks. The proposed approach offers a scalable and low-cost solution for preliminary disease screening, particularly in healthcare environments with limited expert availability. The multi-seed evaluation with confidence intervals provides a rigorous assessment of model stability. Further work is needed to improve label reliability and expand to multi-label classification.
Qing-Yuan Zhang, Pardeep Vasudev, Kezhi Li et al.· Frontiers in Digital Health· 0 citations
Recent progress in Medical Visual Question Answering (VQA) has significantly aided clinical decision-making across various domains such as pneumonia, oncology, and neurology. However, spinal and musculoskeletal ailments remain critically underexplored. The development of reliable VQA models for spinal imaging is currently hindered by a lack of datasets and evaluation protocols that do not reflect the descriptive, diagnostic interpretations used in clinical practice. In this resource paper, we introduce SpineXR-VQA, an open-source, clinically grounded, and verified benchmark comprising 2,187 X-rays and 8,272 expert-verified, open-ended Question-Answer pairs. Unlike standard classification-based datasets, SpineXR-VQA features six expert-validated categories: abnormality, severity, location, diagnosis, treatment, and reasoning. Ten orthopedic specialists from India and Thailand validated all pairs, ensuring geographic diversity and high inter-rater agreement (Cohen's Kappa: 0.96 for questions, 0.93 for answers). We benchmark 15 state-of-the-art Multimodal Large Language Models (MLLMs), including proprietary systems such as Claude Sonnet, GPT-o4 mini, and Gemini Flash, alongside medical-specific and general open-weight models. While these models achieve moderate semantic alignment (median similarity: 0.72), a detailed analysis reveals that they consistently fail to capture essential diagnostic nuances, such as anatomical fidelity and clinical completeness. These results underscore the necessity for specialized models in spinal VQA, a gap SpineXR-VQA fills.
Deepali Mishra, Dr.Sorayouth Chumnanvej, Vikas Trivedi et al.· ACM Transactions on Computin...· 0 citations
Background: Cone-beam computed tomography (CBCT) frequently captures the maxillary sinuses incidentally, and reliable automated detection of sinus abnormality is clinically relevant. Unlike most vision-language benchmarks in medical imaging, which pair images with pre-existing, human-authored clinical reports, findings text can also be generated directly by a large language model from the image itself--raising the question of how much diagnostic value such AI-derived text carries, and whether that value depends on independent verification. Multimodal artificial intelligence (AI) benchmarks risk overstating performance if the provenance of each input--image, raw AI-generated text, or radiologist-verified text--is not clearly separated and reported. Methods: We used 300 mid-sagittal CBCT slices from the MMDental dataset. ChatGPT generated findings text and a provisional normal/abnormal label for every slice (majority vote, three independent readings from the image alone); primary classification performance was assessed on this full, unfiltered set (n=300). A radiologist then independently reviewed each case's image together with ChatGPT's description, producing their own diagnosis; three cases were excluded as insufficient, yielding 297 confirmed cases. On this subset, every model was retrained and re-evaluated under identical 10-fold cross-validation on both the provisional ChatGPT-only labels ("pre") and the radiologist-confirmed labels ("post"), isolating the effect of label provenance from image or architecture. Eight vision architectures, seven language classifiers, and five VLMs were evaluated throughout; three generative models performed exploratory note-drafting. Findings: Raw ChatGPT-generated text produced the highest performance of any modality or condition: language models reached near-ceiling AUC (0.992 to 1.000, n=300), exceeding every vision model (AUC 0.799 to 0.880) and every VLM image-only probe (AUC 0.63 to 0.69). On the 297-case pre/post analysis, this advantage depended heavily on label source: language and text-derived VLM performance fell substantially from ChatGPT-only to radiologist-confirmed labels (e.g. BERT-base AUC 0.999 to 0.837), while vision-model performance was stable or modestly improved (e.g. DenseNet-121 0.867 to 0.891). The radiologist reclassified 62 of 297 cases (21%) relative to ChatGPT's provisional read, and a meaningful proportion of raw ChatGPT text was clinically uninterpretable or unsupported by the imaging. Interpretation: As shown here for the first time, raw, image-derived AI-generated text yields the highest apparent classification performance in this benchmark, but this reflects the text's alignment with its own self-generated labels rather than verified diagnostic content, and a substantial share of that text is not clinically explainable. Radiologist-confirmed text and labels give a lower but trustworthy estimate of true performance, on which convolutional neural network (CNN) vision models remain a stable, comparatively inexpensive baseline. Multimodal dental AI should report performance separately by modality and label provenance rather than pooling headline metrics.
S. Alhebshi, H. Khalifa, T. D. Pham· medRxiv· 0 citations
Chest X-ray imaging remains the most widely used radiological modality for pneumonia screening. While recent advances in deep learning have demonstrated strong diagnostic performance, the deployment of such models in real-world settings requires not only high accuracy but also robustness and interpretability across different model designs and populations. In this work, we present a comprehensive benchmarking study of multiple deep learning architectures for pneumonia detection. A distinctive methodological feature of this study is the use of two demographically distinct datasets: a pediatric chest X-ray dataset for training, and an independent adult population dataset for external validation. All models were evaluated under a standardized cross-validated protocol. Beyond predictive metrics, we conduct an extensive eXplainable Artificial Intelligence (XAI) analysis assessing both qualitative and quantitative properties, including explanation stability and localization fidelity. Results show that model design choices significantly influence both predictive performance and explanation behavior. In particular, certain architectures consistently achieved superior performance while producing more focused and stable explanations. The performance-interpretability ranking is preserved under external validation on a demographically distinct adult cohort, providing evidence that performance-interpretability coupling is robust to population shift and supports the generalizability of the proposed framework beyond the pediatric training distribution. This work contributes a reproducible, end-to-end methodology that jointly optimizes performance and interpretability, offering practical guidance for selecting and deploying explainable deep learning models in clinical pneumonia screening. All code and data are publicly available at: https://github.com/SynergIA-Lab/pneumoniacnn
Francisco A. Gómez-Vela, Aurelio López-Fernández, F. Divina et al.· Applied intelligence (Boston...· 0 citations
Lung disease remains a major global health concern, and accurate diagnosis using chest X-ray images plays a crucial role in supporting effective clinical decision-making. The contribution of this work lies in empirically demonstrating how internal redundancy removal through standard magnitude-based pruning can improve both performance and stability of an established CNN architecture. The COVID-Qu-Ex dataset was utilized, consisting of 11,956 COVID, 11,263 pneumonia, and 10,701 normal X-ray images. All models were trained under identical preprocessing, augmentation, and evaluation protocols, with three different random seeds to ensure result stability and reproducibility. Experimental results show that the pruned InceptionV3 model achieved superior performance, with an accuracy of 95.54% ± 0.0041, precision of 95.62% ± 0.0039, recall of 95.54% ± 0.0041, and F1-score of 95.54% ± 0.0041, outperforming the baseline InceptionV3 as well as other modern CNN architectures such as ResNet101 and VGG19. These findings demonstrate that network pruning can effectively reduce model redundancy while maintaining, and even improving, classification performance. Future work will focus on extending the proposed approach to other CNN architectures, integrating interpretability techniques, and addressing data imbalance issues to enhance clinical reliability.
Joshua Pinem, Widi Astuti, A. Adiwijaya· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
