CEM MoE: A Lightweight Sparse Mixture of Experts Framework for Empathetic Dialogue Generation

Abstract

Empathetic dialogue generation requires models to understand user emotions and produce fluent, relevant, and emotionally appropriate responses. Existing lightweight models offer favorable deployment efficiency, but shallow emotion prediction layers limit their ability to represent fine-grained and dynamically changing emotional states. To address this limitation, CEM-MoE, a lightweight sparse mixture of experts (MoE) framework based on the Commonsense-aware Empathetic Chatting Machine (CEM), is proposed. Unlike the original CEM, CEM-MoE employs two independently routed sparse MoE streams to separately model emotional information from dialogue history and the current utterance. A bounded density-aware fusion mechanism integrates the two emotion distributions, where normalized utterance length serves as a lightweight density proxy and a sigmoid gate constrains the fusion coefficient to a valid range. Auxiliary load balancing regularization and label smoothing further improve expert utilization and generation stability. Experiments on the EmpatheticDialogues and ESConv datasets show that CEM-MoE achieves competitive generation performance with only marginal additional overhead. Compared with CEM, CEM-MoE reduces PPL from 37.76 to 36.59 and improves emotion accuracy from 32.10% to 35.47%, while increasing the parameter count by only 0.12M. Expert activation analysis reveals no obvious expert collapse, and human evaluation indicates higher scores in empathy, fluency, and relevance. The results verify that the dual-stream sparse MoE with density-aware fusion is effective for lightweight fine-grained empathetic dialogue modeling. The source code is publicly available at https://github.com/ZhinanGou/CEM-MoE.