This work proposes a simple, parameter-free intervention that initializes SID token embeddings directly from their corresponding centroids in the semantic embedding space, and shows that preserving SID geometry, beyond shared-prefix structure, provides a simple and effective semantic prior for LLM-based GR.
Abstract
Recent advances in generative recommendation (GR) leverage large language models (LLMs) as recommender backbones, enabling LLMs to directly generate recommendations conditioned on item-interaction histories. In these systems, items are often represented through semantic IDs (SIDs) added to the LLM vocabulary as special tokens. Ideally, SIDs imbue item token representations with semantic priors, thereby improving model generalization. However, standard vocabulary expansion typically initializes these tokens as random Gaussian vectors, discarding the SIDs'underlying continuous geometry and forcing the LLM to relearn token relationships from interaction data. To demonstrate the consequences of this design, we first show that training from this initialization tends to organize SID embeddings around item popularity rather than semantics. We further show that, despite partially reducing the reliance on popularity and improving cold item performance, the computationally expensive process of continual pretraining (CPT) fails to reliably recover the original semantic geometry. To address these findings, we propose a simple, parameter-free intervention that initializes SID token embeddings directly from their corresponding centroids in the semantic embedding space. Requiring only a few lines of code and no additional training or inference overhead, this drop-in approach improves pure-SFT Recall@5 by up to 16%, reaches peak performance with up to 40% fewer SFT steps, and improves cold-item Recall@5 by up to 60%. Moreover, on datasets that benefit from additional CPT, centroid initialization reaches comparable performance while requiring half as many CPT epochs. Together, our findings show that preserving SID geometry, beyond shared-prefix structure, provides a simple and effective semantic prior for LLM-based GR.
The design and launch of SnapLGR, an LLM-based generative retrieval system for short-video recommendation at Snapchat shows that successful production SnapLGR requires joint design across representation learning, vocabulary grounding, and efficient training and serving.
Liam Collins, Jiwen Ren, Donald Loveland et al.· arXiv.org· 0 citations
Large Language Models (LLMs) have demonstrated remarkable general-purpose abilities across a wide range of domains, and these strengths have also been increasingly evidenced in recommender systems. However, existing methods that attempt to integrate collaborative signals into LLMs often fail to preserve their foundational knowledge. This loss is critical in text-rich recommendation, where robust semantic understanding is required to interpret user reviews and item profiles. We propose PALRec, a parameter-preserving augmentation framework that equips an LLM with recommendation capabilities while keeping its original parameters fixed. We first construct evidence-grounded user and item profiles from reviews and use them as concise pseudo-labels for reconstruction. We then introduce lightweight, trainable user and item embedding modules optimized with a multi-task objective that combines next-item prediction and profile reconstruction. These modules are trained jointly to align collaborative signals with the LLM’s semantic space without modifying the backbone. We also employ token-aware loss decomposition and frequency-aware reweighting to stabilize training and mitigate popularity bias. Experiments on public benchmarks show that PALRec consistently outperforms fully fine-tuned counterparts in recommendation accuracy while preserving the LLM’s pre-trained knowledge. This result highlights that maintaining the LLM’s semantic understanding is crucial for effectively exploiting textual information in recommender systems.
Hyunsoo Na, Minseok Gang, Sang-goo Lee et al.· ACM Transactions on Informat...· 0 citations
Item tokenizer encodes semantic embeddings into token IDs to replace the randomly assigned item IDs used in traditional recommendation models, fundamentally addressing the problems of excessive parameters and cold starts. However, the most common tokenizer, RQ-VAE, suffers from low decoding efficiency due to the inherent dependencies among its codebooks. Meanwhile, efficient independent tokenizers such as optimized product quantization (OPQ) still struggle with dimensional correlations and distribution complexity of semantic embeddings. In this work, we propose a f\underline{low}-based item \underline{T}okenizer (Tlow) to transform raw semantic embeddings into a latent space where embeddings conform to a unified standard normal distribution, achieving dual advantages of dimensional independence and distributional simplicity. Independent tokenization performed on these latent embeddings yields semantically clear token IDs. Additionally, we introduce a novel codebook guidance to align the codebook space with the token embedding space, further aiding the learning of more semantically distinct token embeddings. Offline experiments on four public datasets demonstrate that Tlow's tokenization and codebook guidance significantly improve recommendation performance. The improvement on cross-domain and multi-modal recommendations also proves the effectiveness of item tokenization in a simplified embedding space. Online experiments for a multi-modal retrieval task on China's largest social media platform WeChat validate Tlow's powerful distribution transformation capability. The retrieval model based on token IDs improves user CTR by 10.32\% globally and by 11.64\% for new items. Our codes are available at https://github.com/wjjln/Tlow.
Nian Li, Chonggang Song, Jingtao Ding et al.· 0 citations
In generative recommender systems, items are typically tokenized into fixed-length semantic ID sequences for autoregressive next-item prediction. However, for user-context modeling, this fine-grained representation triggers Intra-item Attention Overload: excessive attention is spent on low-level intra-item dependencies rather than high-level inter-item behavioral transitions. To address this, we propose Semantic Subword Tokenization (SST), which represents historical items as variable-length semantic subwords while preserving fixed-length target decoding. SST first applies Item-level Subword Tokenization (IST) to merge stable adjacent atom tokens into compact semantic subword tokens, thereby reducing intra-item reassembly in the encoder. It then introduces Behavior-induced Co-occurrence Augmentation (BCA) to inject coarse-grained semantic prefix transition signals, guiding the freed modeling capacity toward inter-item behavioral regularities. Extensive experiments on three public datasets and three generative recommender backbones show empirical improvements of SST over fixed-length and transferable variable-length SID baselines. Code is available at https://github.com/mxrcandy/Semantic-Subword-Tokenization.
Xinrui Miao, Mingjia Yin, Jiaqing Zhang et al.· 0 citations
Leveraging Large Language Models (LLMs) for generative recommendation has attracted significant research interest, where item tokenization is a critical step. It involves assigning item identifiers for LLMs to encode user history and generate the next item. Existing approaches leverage either token-sequence identifiers, representing items as discrete token sequences, or single-token identifiers, using ID or semantic embeddings. Token-sequence identifiers face issues such as the local optima problem in beam search and low generation efficiency due to step-by-step generation. In contrast, single-token identifiers fail to capture rich semantics or encode Collaborative Filtering (CF) information, resulting in suboptimal performance. To address these issues, we propose three fundamental principles for item identifier design: 1) integrating both CF and semantic information to fully capture multi-dimensional item information, 2) designing order-agnostic identifiers without token dependency, mitigating the local optima issue and achieving simultaneous generation for generation efficiency, and 3) disentangling semantics across different tokens, unlocking the potential of scaling order-agnostic identifier. Accordingly, we introduce a novel set identifier paradigm, representing each item as a set of order-agnostic tokens. To implement, we propose SETRec, which leverages CF and semantic tokenizers to obtain order-agnostic multi-dimensional tokens. To eliminate token dependency, SETRec uses a sparse attention mask for user history encoding and a query-guided generation mechanism for simultaneous token generation. We instantiate SETRec on T5 and Qwen (from 1.5B to 7B). To reinforce disentanglement between tokens, we propose SETRec++, which introduces two disentanglement strategies, i.e., disentangled regularization loss and token masking mechanism. Experiments on four datasets demonstrate its effectiveness across various scenarios (e.g., full ranking, warm- and cold-start ranking, and various item popularity groups). Moreover, results validate SETRec’s superior efficiency and scalability on cold-start items as model sizes increase, and SETRec++’s potential in the scalability of order-agnostic identifier.
Xinyu Lin, Chuan-Bo Zhang, Yu-Fan Liu et al.· ACM Transactions on Recommen...· 0 citations
Generative recommendation commonly represents items using fixed-length semantic identifiers (SIDs) constructed through clustering and quantization. However, these artificial codes may overcompress item semantics, remain misaligned with pretrained LLM vocabularies, and require costly autoregressive decoding. In light of this, we propose VaLiDRec, a generative recommendation framework based on variable-length, LLM-aligned semantic identifiers. VaLiDRec constructs SIDs directly from informative native LLM vocabulary tokens via token importance estimation, semantic-quality-aware pruning, and collision-aware refinement, allowing identifier lengths to adapt to item semantic complexity. To model user preferences, VaLiDRec incorporates graph-aware soft prompts and reformulates recommendation as token-set prediction with token-level item scoring, eliminating autoregressive SID generation and beam search. Experiments on four real-world datasets show that VaLiDRec consistently outperforms strong sequential and generative recommendation baselines across all evaluation metrics. It further achieves superior zero-shot item cold-start performance and 87.49$\times$ faster inference than LC-Rec. These results demonstrate that LLM-native variable-length semantic identifiers provide a more expressive and efficient paradigm for generative recommendation.
