TransRetrieval is presented, a Transformer-based retrieval framework that scales with both computational budget and cross-domain data and introduces weighted average aggregation, which restores the homogeneous-token assumption Transformers rely on, and target token compression that cuts per-candidate FLOPs while preserving cross-attention expressiveness.
Abstract
Applying scaling laws to recommendation retrieval is hindered by feature heterogeneity: naively stacking Transformer layers yields diminishing returns because heterogeneous fields produce severe token-norm divergence. We present TransRetrieval, a Transformer-based retrieval framework that scales with both computational budget and cross-domain data. The key enabler is (1) weighted average aggregation, which restores the homogeneous-token assumption Transformers rely on. Building on this, we introduce (2) target token compression that cuts per-candidate FLOPs by 85% while preserving cross-attention expressiveness, and (3) position-style domain embeddings that unify multiple domains at negligible additional cost, turning cross-domain data into a scaling asset. On a 40-billion-interaction industrial dataset and the public KuaiRand benchmark, scaling compute from 0.1 to 2 MFLOPs per target yields +19.3/+22.2 pt Recall@2000, confirming robust log-linear scaling. In online A/B tests, TransRetrieval lifts platform revenue by 2.53% under the same end-to-end latency constraint as the production baseline.
Scaling Transformers has driven large gains in language modeling, but transplanting this to behavior-sequence modeling in production ranking is challenging: recommendation differs in signal quality, where behavior sequences are noisy, temporally irregular, and sparsely supervised, and in computation asymmetry, where each request scores many candidates against one shared user history under tight latency budgets. We propose ReST, a recommendation-native Transformer scaling framework. For signal quality, it introduces a sequence encoder with dual-gated attention, rotary positional and temporal embedding, stabilized residual normalization, and training-only auxiliary objectives. For computation asymmetry, it factorizes ranking into a heavy reusable encoder and a lightweight cross decoder with projection-free KV attention and token-specific parameterization, coupling user-level shared-prefix training with shared-prefix serving for compute-once, decode-many-times ranking. Across industrial and public benchmarks, ReST achieves higher accuracy and scales more consistently along sequence length, depth, and width, where LLM-style Transformer blocks saturate. A one-week online A/B test on a production advertising platform improves online AUC by 1.31% and lifts a core revenue metric by 11.93% within a 50 ms P99 budget; ReST has since been fully deployed in production, showing that behavior-sequence scaling remains a promising, under-exploited axis for production ranking.
Jie Chen, Xiang-Qian Yu, Yan-Chao Lian et al.· 0 citations
In large-scale industrial search and ranking systems, Click-Through Rate (CTR) prediction is undergoing a paradigm shift from traditional Deep Learning Recommendation Models (DLRM) toward unified, compute-intensive Transformer architectures. The primary motivation for this transition is to leverage Model FLOPs Utilization (MFU) to achieve predictable performance gains through Scaling Laws. However, existing scaling approaches like OneTrans and Climber, often adopt an all-in-tokenization strategy when directly migrating Large Language Model (LLM) architectures, which neglects the unique feature heterogeneity. We propose TmallGS, a high-performance, scalable universal ranking architecture tailored for the Tmall precision ranking domain. TmallGS introduces five core innovations: (1) Hierarchical Distribution-Calibrated Tokenization: To bridge the heterogeneity gap, we propose a coarse-to-fine pipeline combining Field-wise Saliency Reweighting (FSR) and Distribution-Calibrated Projection (DCP) to project diverse features into optimized subspaces. (2) Field-Adaptive Gated Transformer Backbone: We employ Per-Field QKV projections and a noise-adaptive gating mechanism to refine semantic interactions and suppress element-wise noise. (3) Decoupled FiLM Late Fusion: To preserve high-frequency explicit signals, we utilize Feature-wise Linear Modulation (FiLM) to dynamically modulate backbone embeddings with explicit cross-features. (4) Context-Aware Bias Decoupling: Addressing systemic biases beyond position, we incorporate a Context-Aware Bias Net that leverages deep global context to orthogonally decouple bias factors from genuine user intent. (5) Error-Aware Progressive Training: We propose a dynamically weighted loss function based on hierarchical prediction errors, which enables adaptive hard-sample mining to improve model robustness. Extensive offline experiments and online A/B tests conducted in the Tmall. Tmall is China's largest B2C e-commerce platform. Search Ranking stage demonstrate that TmallGS significantly boosts training throughput while delivering substantial gains in both UCTCVR and GMV metrics.
Zhentao Song, Yufeng Gao, Xingye Fang et al.· Proceedings of the 32nd ACM...· 0 citations
Transformer-style architectures are increasingly adopted for industrial recommendation systems, yet they inherit a design premise misaligned with the task: generative models rely on per-token autoregressive prediction, which justifies maintaining large intermediate tensors that scale with sequence length. In contrast, recommendation systems produce a single set of relevance scores for eachpair without token-level supervision. Leveraging this observation, we propose SlimPer, which reformulates personalized ranking as iterative refinement of a compact, unifiedknowledge base. At each layer, the model selectively queries raw multi-modal user-side tokens, computes explicit relevance matching scores, and refines the knowledge base, all in O(N) per-layer cost with a fixed-size intermediate representation. As a result, model depth is decoupled from user history length, enabling deeper relevance understanding without proportional growth in compute or memory; request-only optimization further trims memory by sharing a single copy of user-side tokens across all candidate items. SlimPer unifies sparse, dense, and sequence features within a single backbone and provides inherent interpretability through its attention mechanism. Deployed on Instagram Reels and Feed, SlimPer yields measurable improvements in user engagement while streamlining the overall system and enabling effective modeling of 10k+ fine-grained user history events.
Siqi Wang, Xianjie Chen, Shaofen Deng et al.· arXiv.org· 1 citation
The Fusion of Layer-wise Exits for Sequential Recommendation (FLEXRec), a discriminative framework that enhances compact LLMs while retaining scalable full-corpus ranking and achieves state-of-the-art accuracy among competing methods while remaining highly efficient.
Xurong Liang, Tong Chen, Q. Nguyen et al.· 0 citations
This work proposes REIGN (Refurbished Embeddings with Integrated Guidance Networks), a contrastively trained bi-encoder that operates on sequences of contextualised chunk embeddings from a frozen Guidance Network rather than on raw tokens for document-to-document retrieval.
Large Language Models (LLMs) have demonstrated remarkable general-purpose abilities across a wide range of domains, and these strengths have also been increasingly evidenced in recommender systems. However, existing methods that attempt to integrate collaborative signals into LLMs often fail to preserve their foundational knowledge. This loss is critical in text-rich recommendation, where robust semantic understanding is required to interpret user reviews and item profiles. We propose PALRec, a parameter-preserving augmentation framework that equips an LLM with recommendation capabilities while keeping its original parameters fixed. We first construct evidence-grounded user and item profiles from reviews and use them as concise pseudo-labels for reconstruction. We then introduce lightweight, trainable user and item embedding modules optimized with a multi-task objective that combines next-item prediction and profile reconstruction. These modules are trained jointly to align collaborative signals with the LLM’s semantic space without modifying the backbone. We also employ token-aware loss decomposition and frequency-aware reweighting to stabilize training and mitigate popularity bias. Experiments on public benchmarks show that PALRec consistently outperforms fully fine-tuned counterparts in recommendation accuracy while preserving the LLM’s pre-trained knowledge. This result highlights that maintaining the LLM’s semantic understanding is crucial for effectively exploiting textual information in recommender systems.
Hyunsoo Na, Minseok Gang, Sang-goo Lee et al.· ACM Transactions on Informat...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
