Codebook-Based Semantic IDs in Generative Recommendation: Enabling Interface, Emerging Bottleneck

Abstract

Codebook-based semantic IDs (SIDs), short discrete code sequences produced by quantization over item representations, made generative recommendation practical by turning catalog-scale retrieval into low-cardinality generation. Yet the same interface now concentrates the field’s hardest questions. We revisit the canonical two-stage pipeline for SID-based recommendation — item tokenization using RQ-VAE, K-means variants, or related methods, followed by training a generator to emit the resulting codes — as a past–present–future story. Our thesis is that these identifiers succeeded as an enabling interface, but not yet as a stable theory of item identity for recommendation: a single SID must simultaneously preserve behaviorally useful similarity, protect identity under head-tail skew, survive catalog drift, align with language-model generators, and remain decodable under production constraints. We organize the present literature around five recurring fractures — objective mismatch, identity-versus-sharing tension, static codes in dynamic settings, LLM alignment tax, and serving feedback into identifier design — and argue that SID utility is regime-dependent: model capacity, catalog scale, and interaction density jointly determine whether semantic structure helps or is redundant. We close with a research agenda centered on recommendation-native tokenization, adaptive identity, explicit alignment protocols, and co-design of tokenization with serving.