Beyond End-to-End: A Low-Latency, Retrieval-Augmented Chain-of-Thought for Scalable and Explainable Industrial Recommendation
Abstract
Large Language Models (LLMs) can provide semantic reasoning for recommendation, but direct LLM invocation is incompatible with the latency, stability, and reliability requirements of industrial ranking systems. We propose RA-CoT (Retrieval-Augmented Chain-of-Thought), a deployment-oriented framework that uses LLM-derived structured reasoning without replacing the production CTR/CVR backbone or placing LLM inference on the blocking serving path. RA-CoT asynchronously generates schema-constrained CoT-derived representations from pre-impression behavior summaries, caches them with validity checks, and falls back to the backbone when no valid reasoning vector is available. Valid reasoning is injected through an availability-aware gated residual module as bounded logit-space compensation, making the LLM signal reusable, safely ignorable, and compatible with millisecond-level serving. We evaluate RA-CoT on a large-scale industrial advertising dataset and a public recommendation benchmark, and deploy it in a two-week online A/B test. RA-CoT yields modest but consistent offline ranking improvements across DNN, DIEN, BST, and SIM backbones. In production, it improves CTR by 2.48%, CVR by 1.95%, and RPM by 2.12%, while keeping P99 serving latency at 54 ms. These results show a practical path for using cached and bounded LLM-derived structured reasoning in industrial recommender systems.