Evidential Federated Learning for Robust Multimodal Emotion Recognition in Conversations

Abstract

Emotion Recognition in Conversations (ERC) at scale faces two coupled obstacles: conversational data is partitioned across organizations under privacy constraints, and emotion labels are costly, leaving most utterances unlabelled. Federated learning addresses the first, but standard aggregation trusts every client update equally, and federated semi-supervised methods inherit the confirmation bias of hard pseudo-label thresholds. We propose a backbone-agnostic framework that threads one evidential uncertainty estimate through three mechanisms: an aggregation rule (EAFA) that re-weights clients by a single privacy-preserving quality scalar; a threshold-free consistency loss (ECR) whose gradient vanishes when the model is uncertain; and an evidence-level fusion (ELF) that degrades gracefully when audio is missing. A central and deliberately honest finding, established on both real corpora and a controlled federated-EDL testbed built and executed in full, is that uncertainty weighting alone is not a robust aggregator: its gains under symmetric label noise are marginal and statistically non-significant, and it is defeated by update-poisoning adversaries that report low uncertainty. We therefore add EAFA-Guard, which couples the evidential weight with a server-root direction filter and a magnitude cap. On the testbed EAFA-Guard is the strongest aggregator we test against label-flip, sign-flip, and adaptive attacks, holding about 89% macro-F1 where Multi-Krum and Coordinate-Median each fail on at least one attack, and about 94% as malicious clients rise to 40%, where the baselines fall below 56%. On MELD, IEMOCAP, and DailyDialog the framework is competitive while our calibration analysis exposes a genuine underconfidence limitation. Every reported number derives from executed code.