Auditing Alignment among Item Relevance, Goal Progress, and LLM-Judged Responses in Music-CRS

Abstract

The official evaluation of the RecSys Challenge 2026 Music-CRS task can disagree with itself. The task scores a ranked list of 20 tracks and a natural-language response per turn with a single composite of nDCG@20, catalog and lexical diversity, and an LLM judge. Auditing the released training data and the interim leaderboard, we find two internal conflicts. First, the accuracy target contradicts the dataset’s own goal labels. When a listener explicitly asks for a different artist, the next ground-truth track still stays with an artist the session has already played in 76.0% of cases, 12.9 points more often than on other turns. The pattern sharpens when the dataset’s own goal-progress label marks the preceding same-artist recommendation as unsuccessful. The next track then repeats that very artist in 85.8% of cases. A ranker that follows the user’s request is therefore penalized by nDCG. Second, the response terms of the composite move independently of the recommendations. We froze one top-20 ranking on the blind leaderboard and rewrote only the responses. The composite rose by 0.0843, which equals the contribution of a +0.17 nDCG@20 gain. The composite cannot tell better recommendations from better-worded explanations. We recommend reporting goal adherence and ranking–response consistency alongside it. Team Komekami ranked 8th in the Industry Track and 16th overall.