DuetMoE: Coupling Inter- and Intra-Subgroup Robustness for Fair Medical Image Analysis

Abstract

As medical AI expands across diverse healthcare settings worldwide, equitable performance across patient populations is becoming essential to trustworthy clinical use. Fairness in medical image analysis is often evaluated through average performance across predefined subgroups, yet similar subgroup averages can conceal substantial variation among individual patients. Therefore, a reliable medical AI requires addressing two complementary objectives: \emph{inter-subgroup fairness}, which reduces performance disparities across groups, and \emph{intra-subgroup robustness}, which protects poorly served patients within each group. To jointly address these objectives, we propose \textbf{DuetMoE}, a subgroup-aware mixture-of-experts framework that couples group-level adaptation with patient-specific clinical guidance, enabling more reliable medical image analysis for individual patients. For settings without linked clinical records, we further introduce \textbf{DuetMoE+}, which retains subgroup-routed experts and incorporates a KL-constrained distributionally robust objective to address intra-subgroup robustness. We evaluate our methods on PI-CAI, radiotherapy, and Harvard-FairSeg. Across four evaluation settings, our methods lead in overall or equity-scaled performance, reduce the inter-subgroup mean Dice gap by up to $30.7\%$, and raise intra-subgroup 25th-percentile Dice by up to 11.5 points over the strongest reported baselines.