A Concept-Guided Fusion Framework for Interpretable Skin Lesion Classification

Abstract

Malignant skin lesions are often visually indistinguishable from benign analogs, presenting a challenging "cancer-mimic" scenario. Post-hoc methods, such as Grad-CAM and SHAP, fail to ground model decisions in the ABCD clinical criteria (Asymmetry, Border, Color, Diameter) relied upon by dermatologists. To address this, we propose an auxiliary concept fusion framework: a domain-adapted DINOv2 concept head is trained on segmentation-derived ABCD targets and at inference predicts ABCD scores directly from raw images; its outputs are fused with backbone visual features at classification time, yielding auditable concept co-predictions without imposing a strict information bottleneck. Across five backbones (CoAtNet, EfficientNetV2, MobileNetV3, Swin, ConvNeXt V2) on a four-class "cancer-mimic" task (melanoma, nevus, seborrheic keratosis, basal cell carcinoma), CoAtNet shows the most consistent accuracy and F1 gains. A CoAtNet random-concept control supports that these gains reflect structured concept information rather than added dimensionality. Our findings demonstrate that the utility of concept-integration is architecture-dependent, motivating a backbone-aware fusion design for interpretable dermoscopy AI.