Skip to content
Book Open access

Codebook-Based Semantic IDs in Generative Recommendation: Enabling Interface, Emerging Bottleneck

Sep 2026 · Proceedings of the 20th ACM Conference on Recommender Systems · pp. 801-807 · 3 citations · 20 references

Abstract

Codebook-based semantic IDs (SIDs), short discrete code sequences produced by quantization over item representations, made generative recommendation practical by turning catalog-scale retrieval into low-cardinality generation. Yet the same interface now concentrates the field’s hardest questions. We revisit the canonical two-stage pipeline for SID-based recommendation — item tokenization using RQ-VAE, K-means variants, or related methods, followed by training a generator to emit the resulting codes — as a past–present–future story. Our thesis is that these identifiers succeeded as an enabling interface, but not yet as a stable theory of item identity for recommendation: a single SID must simultaneously preserve behaviorally useful similarity, protect identity under head-tail skew, survive catalog drift, align with language-model generators, and remain decodable under production constraints. We organize the present literature around five recurring fractures — objective mismatch, identity-versus-sharing tension, static codes in dynamic settings, LLM alignment tax, and serving feedback into identifier design — and argue that SID utility is regime-dependent: model capacity, catalog scale, and interaction density jointly determine whether semantic structure helps or is redundant. We close with a research agenda centered on recommendation-native tokenization, adaptive identity, explicit alignment protocols, and co-design of tokenization with serving.

Read PDF

Similar papers

Book Open access Sep 2026

CoSID: Concept-Conditioned Semantic-ID Decoding for Efficient Generative Recommendation

Generative recommenders retrieve items by autoregressively decoding semantic IDs (SIDs). The standard autoregressive SID interface (SID-AR) represents each history item with K code tokens, expanding a T-item history to TK tokens, while trie-constrained beam search repeatedly re-enters the full backbone during generatio...

Danil Gusak, A. Volodkevich, Evgeny Frolov · 3 citations
#artificial intelligence Preprint Sep 2026

FineSID: Scalable and Efficient Semantic Identifier Learning for Generative Recommendation

FineSID is a unified quantization framework that moves beyond Top-1 assignment by enabling fine-grained gradient propagation across the entire codebook, effectively alleviating SID collisions and stabilizing training in large, high-dimensional codebooks.

Song-Li Wu, Wei-Nan Gan, Zhao-Cheng Du et al. · 2 citations
#artificial intelligence Preprint Oct 2026

Beyond Semantic IDs: Finite-Beam Reachability and Continuous-Query Generation in Generative Recommendation

Generative recommendation (GR) of social media platforms commonly retrieves items by autoregressively decoding discrete Semantic IDs (SIDs) with finite-width beam search. This work examines how this output interface constrains whether held-out targets enter finite returned lists, while distinguishing catalog-wide score...

Yu Sun, Xiang Ren, Zuo-Dong Xiang et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Evolving Semantic Indexes for Generative Retrieval

Generative retrieval (GR) casts search and recommendation as autoregressive generation of discrete semantic identifiers (SIDs). SID design largely determines GR quality, but its discrete, interdependent components (SID construction, collision handling, decoding, and training) cannot be optimized end-to-end with gradien...

Jinming Nian, Jyun-Yu Jiang, Henry Tsai et al. · 0 citations
Book Open access Sep 2026

Not All Branches Are Equal: Adaptive Semantic ID Construction for Generative Recommendation

Adaptive Silhouette Tree (AST) is proposed, a top-down divisive clustering method that constructs semantic IDs as paths within a tree structure that significantly outperforms strong baselines on Amazon Review datasets of varying sizes and domains.

Guy Hadad, Haggai Roitman, Bracha Shapira · 0 citations
Preprint Oct 2026

Rethinking Semantic ID Construction for Generative Recommendation: SimHash with Parallel Decoding and Semantic Alignment

Semantic ID-based generative recommendation represents each item as a sequence of discrete tokens, enabling structured modeling of item semantics. A critical challenge is constructing semantic IDs that are both semantically expressive and computationally efficient. While recent approaches favor complex learned quantiza...

Yu-Qing Liu, Hui-Yuan Chen, Yi-Bo Wang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.