Simulating Users to Evaluate LLM-Generated Explanations in Recommender Systems: A Unified Agentic Framework and Empirical Study with Preliminary Results

Abstract

Large Language Models (LLMs) are increasingly used both as recommenders and to explain recommendations, yet producing faithful, user-centric explanations and evaluating them rigorously remain open problems, compounded by a documented reproducibility crisis in recommender systems research. In this work we formalise recommendation as a Partially Observable Markov Decision Process and propose a unified, three-tier agentic framework that couples a simulation environment (RecoWorld), a reproducible policy engine (RecBole), and a cognitive LLM user agent. As an offline, single-turn validation of this framework on the MovieLens-32M dataset, we benchmark preliminary single-epoch checkpoints of traditional policies (Bayesian Personalized Ranking (BPR), Convolutional Neural Collaborative Filtering and Neural Matrix Factorisation (NeuMF)), generate LLM-based explanations from the BPR recommendations and score them with quantitative diversity, faithfulness, counterfactual and readability metrics over roughly one million user-item pairs, and run an LLM-as-user interview simulation in which explanations produced by a Qwen2.5 explainer are assessed by a Llama-3 respondent through a six-criterion explanation-quality survey; a controlled comparison on 2000 users shows that replacing BPR with the more accurate NeuMF policy leaves these survey scores practically unchanged. The generated explanations prove highly faithful (feature precision 0.9996, sufficiency 0.9999) and user-tailored explanations (unique-sentence ratio 0.9978), and simulated users rate clarity highest (3.86/5) yet trust lowest (2.60/5). Faithful template explanations are, therefore, trusted less than they are understood, motivating a shift towards controllable generative explanation and scalable simulated evaluation.