Lesion-semantic token guided transformer fusion for stage-adaptive diabetic retinopathy grading and screening

Abstract

Diabetic retinopathy (DR) screening from retinal fundus images is still difficult because of the heterogeneity of lesions, significant class imbalance, and low reliability of existing deep learning algorithms owing to their generic feature pooling and passive fusion mechanisms. To overcome such limitations, this study presents a hybrid model called HandEffiTrans-DR, which combines deep retinal representations with handcrafted lesion descriptors via the directed transformer-based attention mechanism. First, a lesion-prior-guided token squeezer (LPG-TS) module is proposed in the work, which is capable of creating lesion-specific visual tokens via integrating prior knowledge about pathology into the tokenization procedure. Moreover, a directed transformer-based fusion (T→V) strategy allows for actively fusing the handcrafted lesion features with visual representations. The introduced framework is assessed with a stratified five-fold cross-validation protocol. Its average classification accuracy surpasses 97%, while the macro-F1 score is higher than 0.97 and Cohen's kappa is greater than 0.95, indicating good agreement despite the presence of class imbalance. The probability reliability is measured in terms of expected calibration error (ECE) and Brier scores and shows well-calibrated confidence. Further validation is performed on two external datasets, where consistent performance with minor decline is shown.