Skip to content

Data Processing for Offline Evaluation in Recommender Systems: a Survey

Sep 2026 · arXiv.org · Vol abs/2609.31696 · 0 citations
Computer Science

TL;DR

A systematic, cross-domain characterisation of data processing practices for the offline evaluation of recommender systems is provided, identifying substantial heterogeneity in how auxiliary information is prepared and represented, as well as inconsistencies in the specification of data splitting protocols.

Abstract

Offline evaluation is the dominant experimental paradigm in recommender systems research, enabling reproducible and cost-effective comparisons on historical interaction data. Yet, while considerable attention has been devoted to recommendation models and evaluation methodologies, the data processing decisions that precede model training have received less scrutiny. These decisions determine the information available to recommendation algorithms and can affect the comparability and reproducibility of experimental results. This survey provides a systematic, cross-domain characterisation of data processing practices for the offline evaluation of recommender systems. We examine the data-centric pipeline, from dataset selection and interaction representation to data preparation, multimodal feature extraction, and train-validation-test splitting. Our analysis spans recommendation paradigms, including collaborative, sequential, session-based, graph-based, knowledge-aware, context-aware, multimodal, federated, cross-domain, contrastive-learning, and LLM-based recommendation. Beyond reviewing existing practices, we introduce a unified framework and taxonomy for describing data transformations and feature-extraction strategies, distinguishing data preparation from the extraction of representations from multimodal side information. Our empirical analysis reveals a landscape dominated by a narrow set of dataset-level transformations, particularly support-driven filtering, while representation-dependent transformations remain less common. We further identify substantial heterogeneity in how auxiliary information is prepared and represented, as well as inconsistencies in the specification of data splitting protocols, where similar labels may conceal different experimental conditions.

View source

Similar papers

Preprint Sep 2026

FINALLY: A Dataset Recommender System for Recommender-Systems Research

These results establish technical consistency for the evaluated FINALLY workflow and show that the implemented strategies follow their intended optimization directions within the investigated configuration space, but do not establish the scientific suitability, global optimality, or practical superiority of the generat...

L. Owie · 1 citation
Conference Aug 2026

An Efficient SVD-Based Recommendation System with Data Preprocessing and Comprehensive Performance Evaluation

The Recommendation system has now evolved to be an integral part of current e-commerce, media streaming, online learning, and social networking sites where the system recommends suitable products and content according to the past behavior of the user. Conventional collaborative filtering algorithms face issues related...

Kirti Soni, S. Tewari · 0 citations
Review Open access Oct 2026

CASARS Recommender Systems: A Systematic Review on Methods, Datasets, Limitations, and Future Directions

Among the main objectives of recommender systems, the minimization of the problem known as information overload stands out. The personalization of recommendations, which emerge as users interact across various digital platforms, aims to mitigate this recurring issue in the modern world. Over the years, different reco...

M. Leme, V. Simm, D. Tanno et al. · 0 citations
Open access Sep 2026

A Systematic Analysis of Methods for Uncertainty Quantification in Distribution-based Recommender Systems

Recommender systems (RS) have become necessary tools for personalizing content and services across digital platforms, yet their confidence estimation remains a critical concern. Confidence estimation in RS aims to answer how confident the model is in its predictions. Despite the advances, integrating confidence into RS...

J. Pires, E. D. da Silva, Frederico Araújo Durão · 0 citations
Book Open access Sep 2026

A Framework for Value-Aligned Modeling and Robust Experimentation

A systematic framework for aligning machine learning development with core business objectives, leading to more accurate decision-making and significant (2x) improvements in engineering productivity is introduced.

Krishna Harsha Reddy Kothapalli, Angela Liu, Kamal Nayan Reddy Challa et al. · 0 citations
Open access Sep 2026

Designing and Evaluating Calibration-Driven Recommender Systems for Fairness and Explainability

Recommender systems are increasingly required to satisfy fairness and transparency constraints. However, the interaction between calibration-based fairness interventions and recommendation explainability remains insufficiently understood. This paper proposes a calibration-driven framework that extends single-category c...

P. Atauchi, A. L. Zanon, Leonardo Rocha et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.