A Systematic Analysis of Methods for Uncertainty Quantification in Distribution-based Recommender Systems
Abstract
Recommender systems (RS) have become necessary tools for personalizing content and services across digital platforms, yet their confidence estimation remains a critical concern. Confidence estimation in RS aims to answer how confident the model is in its predictions. Despite the advances, integrating confidence into RS models often degrades performance, with outcomes strongly dependent on both dataset characteristics and architectural suitability. To investigate these issues, we conducted a comprehensive empirical study over established distribution-based confidence-aware models, OrdRec, CPMF, CBPMF, LBD, PRGAT, and PRLIGHTGCN. We further performed systematic capacity ablations on deeper models. The methodology combined expected calibration error, confidence-error correlation, rating distribution comparisons, and capacity-vs-performance ablation. Our results show that integrating confidence estimation does not consistently improve and can degrade point-estimate performance in recommendation models. Increasing model capacity fails to yield gains and can negatively impact performance, even in the training set, indicating optimization and data constraints. Moreover, calibration quality is not reliably aligned with predictive accuracy. These models may exhibit meaningful confidence–error correlation while remaining poorly calibrated in absolute terms.