Time-aware Confidence Calibration Analysis in Recommender Systems Methods
Abstract
Recommender Systems (RS) are typically evaluated using accuracy and ranking metrics, but these metrics do not show whether the distribution of predicted ratings resembles the distribution of observed ratings. This aspect is especially important in confidence-aware recommendation, where confidence calibration indicates whether predictions assigned higher confidence are indeed more reliable. This work investigates the relationship among distributional fidelity, predictive performance, and confidence calibration in RS under temporal recommendation settings.We evaluate CPMF, LBD, OrdRec, PRLIGHTGCN, and MC Dropout on the MovieLens 1M dataset using an incremental time-series cross-validation protocol, and analyze the effects ofWasserstein-based distribution regularization. The results show that models produce characteristic prediction distributions that remain relatively stable over time but do not converge toward the empirical rating distribution, even as additional interactions become available. Moreover, strong recommendation accuracy does not necessarily imply statistical fidelity to the observed data. Confidence calibration evolves over time and depends strongly on model architecture, while modifications to the global prediction distribution have limited effects on user-item confidence calibration. Overall, distributional fidelity emerges as a complementary dimension for evaluating trustworthy and confidence-aware recommender systems.