OffTrail: A Benchmark for Natural-Language Context in Recommendation beyond Interaction Histories
Abstract
Recommender systems are increasingly expected to combine interaction histories with language. In many cases, a user's next choice may depend on a temporary situation---such as their current mood, plans, or constraints---that is not visible in past behavior. Existing benchmarks, however, mostly evaluate average next-item accuracy, broad instruction following, or content understanding; they do not isolate recommendation episodes in which external situation context is plausibly needed. We introduce e OffTrail, a benchmark built on MovieLens for situation-aware recommendation beyond interaction histories. It contains two complementary families of episodes: observed off-trail choices that are real user transitions but hard to infer from prior behavior, and neighbor-supported alternative continuations that represent plausible futures for similar recent histories. Each episode packages a user history, target item, item metadata, and generated life-situation context. The contexts are treated as plausible external circumstances rather than ground-truth user intentions. OffTrail supports controlled evaluation across sequential models, LLM-based recommenders, and hybrid systems by keeping the recommendation episode fixed while exposing or withholding the situation context. By releasing the artifacts, construction pipeline, and evaluation protocol, OffTrail provides a reproducible testbed for studying whether natural-language context improves ranking when interaction history is weak or ambiguous. Benchmark: https://doi.org/10.5281/zenodo.20560655.