A systematic, cross-domain characterisation of data processing practices for the offline evaluation of recommender systems is provided, identifying substantial heterogeneity in how auxiliary information is prepared and represented, as well as inconsistencies in the specification of data splitting protocols.
Abstract
Offline evaluation is the dominant experimental paradigm in recommender systems research, enabling reproducible and cost-effective comparisons on historical interaction data. Yet, while considerable attention has been devoted to recommendation models and evaluation methodologies, the data processing decisions that precede model training have received less scrutiny. These decisions determine the information available to recommendation algorithms and can affect the comparability and reproducibility of experimental results. This survey provides a systematic, cross-domain characterisation of data processing practices for the offline evaluation of recommender systems. We examine the data-centric pipeline, from dataset selection and interaction representation to data preparation, multimodal feature extraction, and train-validation-test splitting. Our analysis spans recommendation paradigms, including collaborative, sequential, session-based, graph-based, knowledge-aware, context-aware, multimodal, federated, cross-domain, contrastive-learning, and LLM-based recommendation. Beyond reviewing existing practices, we introduce a unified framework and taxonomy for describing data transformations and feature-extraction strategies, distinguishing data preparation from the extraction of representations from multimodal side information. Our empirical analysis reveals a landscape dominated by a narrow set of dataset-level transformations, particularly support-driven filtering, while representation-dependent transformations remain less common. We further identify substantial heterogeneity in how auxiliary information is prepared and represented, as well as inconsistencies in the specification of data splitting protocols, where similar labels may conceal different experimental conditions.
These results establish technical consistency for the evaluated FINALLY workflow and show that the implemented strategies follow their intended optimization directions within the investigated configuration space, but do not establish the scientific suitability, global optimality, or practical superiority of the generat...
The Recommendation system has now evolved to be an integral part of current e-commerce, media streaming, online learning, and social networking sites where the system recommends suitable products and content according to the past behavior of the user. Conventional collaborative filtering algorithms face issues related...
Kirti Soni, S. Tewari· International Workshop on Ar...· 0 citations
Among the main objectives of recommender systems, the minimization of the problem known as information overload stands out. The personalization of recommendations, which emerge as users interact across various digital platforms, aims to mitigate this recurring issue in the modern world. Over the years, different reco...
M. Leme, V. Simm, D. Tanno et al.· WIREs Data Mining and Knowle...· 0 citations
Recommender systems (RS) have become necessary tools for personalizing content and services across digital platforms, yet their confidence estimation remains a critical concern. Confidence estimation in RS aims to answer how confident the model is in its predictions. Despite the advances, integrating confidence into RS...
J. Pires, E. D. da Silva, Frederico Araújo Durão· Brazilian Symposium on Datab...· 0 citations
A systematic framework for aligning machine learning development with core business objectives, leading to more accurate decision-making and significant (2x) improvements in engineering productivity is introduced.
Krishna Harsha Reddy Kothapalli, Angela Liu, Kamal Nayan Reddy Challa et al.· Proceedings of the 20th ACM...· 0 citations
Recommender systems are increasingly required to satisfy fairness and transparency constraints. However, the interaction between calibration-based fairness interventions and recommendation explainability remains insufficiently understood. This paper proposes a calibration-driven framework that extends single-category c...
P. Atauchi, A. L. Zanon, Leonardo Rocha et al.· Brazilian Symposium on Datab...· 0 citations
Related blog posts
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026