Privileged Self-Distillation (PSD) is proposed, a framework that separates learning-time information from inference-time information and uses an advantage-reachability gate to focus distillation on teacher signals likely supported by the observed prefix, along with a momentum-averaged teacher for stable targets.
Abstract
Sequential recommenders are commonly trained with one-hot next-item labels under a causal (prefix-only) objective aligned with inference. While deployment-compatible, this supervision offers little insight into relative preferences among non-target items. Yet logged interaction sequences contain an additional supervisory source: interactions following the target often reveal how user intent evolves, making the target easier to interpret. We treat these future interactions as training-only privileged information, available during learning but not at inference. This raises a natural question: can future interactions provide richer supervision while keeping training aligned with inference-time prediction? We propose Privileged Self-Distillation (PSD), a framework that separates learning-time information from inference-time information. PSD applies two attention masks to the same backbone: a future-aware view yields a privileged teacher distribution conditioned on past and future interactions, while a prefix-only view yields the student distribution used for deployment. Distilling the privileged distribution converts future interactions into training-only supervision rather than inference-time inputs. Since both views share a backbone, the teacher's advantage is purely informational, not architectural, removing the need for a separately pretrained teacher and letting its supervision adapt as the student evolves. PSD further uses an advantage-reachability gate to focus distillation on teacher signals likely supported by the observed prefix, along with a momentum-averaged teacher for stable targets. The framework is optimized end-to-end in a single stage, leaving the deployed model and inference cost unchanged. Experiments across public benchmarks and diverse backbones show consistent improvements.
Sampling multiple solutions and returning the majority answer is among the most reliable ways to improve the reasoning accuracy of large language models without labels, and a growing family of methods converts this consensus signal into training supervision. However, existing approaches use consensus only in restricted forms: as a filter that selects solutions for fine-tuning, as a preference between answers, or as a scalar reward for reinforcement learning, discarding most of the information that the agreeing solutions contain. We present CANON (Consensus-ANchored self-distillatiON), a label-free training method that turns consensus into dense, token-level supervision. For each unlabeled prompt, CANON samples multiple solutions, extracts the majority answer, and conditions a frozen snapshot of the model on a solution that reaches it; this consensus-anchored teacher then supervises the model on its own rollouts at every token. Experiments on mathematical and scientific reasoning benchmarks show that CANON improves pass@1 by up to 12 points, outperforming label-free reinforcement learning by 6 points at a seventh of its compute and approaching a teacher conditioned on gold solutions; trained on pooled unlabeled data, it transfers to held-out benchmarks, matching training methods that use gold labels. Analysis suggests that the improvements are not pure distribution sharpening: after training, the model solves problems it previously never solved in 32 attempts, and its majority vote itself becomes more accurate.
John Gkountouras, Josip Jukic, Ivan Titov· arXiv.org· 2 citations
Generative recommendation formulates next-item prediction as conditional autoregressive generation over discrete Semantic IDs, enabling end-to-end recommendation over large-scale item spaces. However, most existing methods follow a history-as-context paradigm that repeatedly reconstructs user preference from behavior history while discarding system-side recommendation decisions after each request. This creates an asymmetric memory: the system remembers what the user has done, but not what it has previously recommended or learned from the resulting feedback. Consequently, useful preference-validation signals, potential negative evidence, and historical exploration information cannot be directly reused across requests. To address these limitations, we propose LoopMemGR, a closed-loop recommendation experience memory framework for generative recommendation. In addition to the conventional behavior log, LoopMemGR maintains a recommendation experience log that records past recommendation--feedback trajectories. It extracts request-relevant evidence through three complementary views: the recency view captures short-term interaction dynamics, the frequency view summarizes recurring recommendation patterns, and the global view distills transferable regularities shared across users. These signals are compressed into a fixed number of experience tokens to condition the generative backbone under a bounded input budget. Extensive experiments on an industrial Taobao dataset demonstrate the effectiveness of closed-loop experience accumulation and multi-view experience extraction.
Hui Qian, Chang-Fa Wu, Chang Liu et al.· arXiv.org· 0 citations
Interest changes complicate sequential recommendation when interaction histories are combined with item content. We evaluate MM-DRLSR on four public Amazon and Yelp benchmarks. The model integrates category-overlap drift supervision, history-derived drift representations, lightweight identifier–image–text fusion, candidate-conditioned scoring, and an offline actor–critic surrogate objective trained by logged-context replay. The observed next item is used only to construct training labels and rewards; inference ranks candidates from the observed history and candidate content. In the reported five-run summaries under a common leave-one-out protocol, MM-DRLSR attains the highest mean Recall and NDCG among the evaluated methods, with consistent advantages of 0.12–0.22 percentage points over the strongest contemporary multimodal baselines (relative gains of about 1.1–3.6%) that reach Holm-adjusted significance on three of four metrics against Harnessing MLLMs and on Recall@20 against DMESR. The practical value of the method lies in reaching, and in several comparisons, significantly exceeding, the accuracy of heavyweight multimodal-LLM-style approaches with a lightweight architecture whose inference overhead is only about 20% above IDURL. Ablation, sensitivity, and observed interest-shift summaries further describe the contributions of multimodal fusion and offline policy adaptation. The reported results indicate competitive public-data sequential ranking under the stated protocol, together with a reproducible and inference-safe evaluation design.
Changcheng Shao, Cheng Zeng, Xiao-Gang Ye et al.· Applied Sciences· 0 citations
Semantic-ID-based generative recommenders enable efficient next-item generation, but their item-level supervision mainly captures behavioral co-occurrence and local transitions. Large language models (LLMs) can complement these models by reasoning over heterogeneous interaction histories to understand the user's current demand. However, LLMs are not inherently trained with recommendation-specific outcome feedback, and linguistically plausible reasoning therefore does not necessarily lead to effective recommendation decisions. We term this mismatch the Understanding-Action Gap. Accordingly, we distinguish intent knowledge, which captures the user's current demand, from policy knowledge, which specifies the recommendation direction and rejection boundary under that demand. To bridge this gap, we propose a feedback-driven agent framework that first induces task-oriented intent and then discovers recommendation policies according to their incremental utility over an intent-only baseline. Candidate policies are evaluated and refined using outcome-derived feedback rather than linguistic plausibility. We further transfer the resulting intent and policy knowledge into two latent tokens of a lightweight Semantic-ID generator through dual-space relational distillation, enabling LLM-free online inference. Experiments on public benchmarks show consistent improvements over baselines, while large-scale online A/B tests achieve gains of 4.506% in Revenue and 4.621% in ADVV.
Z. Chen, Minmao Wang, Xingchen Liu et al.· arXiv.org· 0 citations
Large Language Models (LLMs) have recently emerged as powerful backbones for recommendation. To better elicit their capabilities, reasoning has been widely incorporated to help LLMs interpret rich textual signals and improve recommendation accuracy. However, explicitly generating intermediate reasoning traces often incurs substantial computational costs, which limits practical deployment in real-world recommender systems. To address this challenge, we propose SelfDR, a Self-Distillation from Reasoning framework for LLM-based Recommendation. SelfDR distills an LLM's own reasoning-enhanced predictions to produce recommendations directly, improving recommendation effectiveness while maintaining inference efficiency. All components in the framework are built on the same base LLM, without relying on any external models. Specifically, the teacher recommender is constructed by training a reasoner with downstream performance as the reward, enabling it to generate targeted rationales that are later incorporated into the teacher's input. A student recommender for direct recommendation, with the same underlying model, then learns from the teacher through self-distillation with a dynamic weighting strategy. Extensive experiments on three public datasets validate the effectiveness, rationality, and efficiency of SelfDR. Codes are available at https://github.com/JiangDeccc/SelfDistillation.
Chumeng Jiang, Jiayin Wang, Xin-Jie Lin et al.· 0 citations
Anonymous short-session recommendation aims to predict the next item from only a few interactions in the current session, without relying on persistent user identities or long-term preference profiles. This setting is particularly challenging because the observed behavioral evidence is often sparse, multiple latent intents may coexist within the same short session, and useful cross-session historical signals are not always effectively exploited. As a result, models that rely mainly on a deterministic session representation often struggle to distinguish between several equally plausible intent explanations under limited observations. To address these issues, we propose RIDRec, a retrieval-enhanced probabilistic framework for anonymous short-session recommendation. Specifically, RIDRec first retrieves relevant cross-session contexts to construct retrieval-conditioned context features and prior-conditioning signals that complement sparse in-session evidence. It then models session intent as a latent distribution with Neural Processes, allowing the model to preserve intent uncertainty and capture multiple plausible interests without predefining the number of intents. On top of this, RIDRec further introduces a conditional diffusion module to progressively refine latent interest representations, making them more robust and discriminative for final ranking. We conduct experiments on multiple public benchmarks under a unified full-ranking evaluation protocol. The results show that RIDRec consistently outperforms strong baselines in both ranking accuracy and item coverage. Moreover, under a 20% training-data setting, RIDRec remains competitive and still surpasses the strongest full-data baseline on RetailRocket in terms of HR@10. Overall, RIDRec provides an effective unified framework for anonymous short-session recommendation. Code is available at RIDRec.
Peilin Liu, Zhiquan Ji, Gang Yan· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
