Preprint

PromptPack: Scaling LLM Annotation Agents for Online Recommendation

Sebastian Koralewski Merwan Barlier Yulia Stolin Blaž Škrlj

Jul 2026 · 0 citations · 15 references

Computer Science

TL;DR

This work introduces PromptPack, a scalable, high-throughput LLM annotation agent that cuts LLM costs by 89% and accelerates throughput by 2.5x while fully preserving AUC.

Abstract

Online recommendation platforms increasingly use Large Language Models (LLMs) to extract structured features from ad creatives. While deploying a single-call LLM annotation agent yields significant Click-Through Rate (CTR) improvements in our live production environment, per-creative prompting is prohibitively expensive to scale. The redundant system instructions sent in every request account for 94% of billed input tokens. To break this cost bottleneck, we introduce PromptPack, a scalable, high-throughput LLM annotation agent. PromptPack achieves this scale via in-context batching, combining a shared system prompt, a strict XML structural envelope, and an output correction layer to ensure deterministic, pipeline-ready feature extraction across multiple creatives simultaneously. We evaluate PromptPack via an offline retrieval benchmark using a downstream logistic-regression ranker. To deeply profile the agent's behavior, we measure AUC and introduce Volume-Weighted Absolute Lift (VWAL), a novel metric capturing the signal quality of the generated features. Compared to our live, unbatched production baseline, PromptPack at batch size 20 cuts our LLM costs by 89% and accelerates throughput by 2.5x while fully preserving AUC.

arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation

A sequential behavioral alignment framework pairing fine-tuning with preference optimization over paired correct and counterfactual rationales is developed and applied, demonstrating that behavioral alignment mitigates bidirectional rationalization while delivering human-interpretable reasoning traces without manual pi...

Alireza S. Ziabari, Kat Ellis, Colleen E. Chan et al. · 0 citations

Jul 2026

LLM-Based Generative Retrieval for Snapchat Content Recommendation

The design and launch of SnapLGR, an LLM-based generative retrieval system for short-video recommendation at Snapchat shows that successful production SnapLGR requires joint design across representation learning, vocabulary grounding, and efficient training and serving.

Liam Collins, Jiwen Ren, Donald Loveland et al. · arXiv.org · 0 citations

Preprint Jul 2026

Prompt Generation Technical Report

Generative retrieval has become an increasingly adopted paradigm for industrial search, recommendation, and advertising systems, delivering significant online gains. Most existing work combines user behavior sequences with large language models (LLMs) to model user preferences. In practice, feature engineering remains...

Dan Ou, Gui Ling, Haokai Wan et al. · 0 citations

Preprint Aug 2026

TSPORec: Token Selection via Preference Optimization for LLM-Based Sequential Recommendation

This work proposes a novel Token Selection approach for Preference Optimization in LLM-based sequential Recommendation, i.e., TSPORec, which accurately pinpoints informative tokens throughout the entire textual content to improve recommendation performance.

Wenqiao Zhu, Chao Xu, Haipang Wu et al. · 0 citations

Jul 2026

Tokenizing Numerical and Embedding Features for LLM RecSys

Experiments on three Amazon recommendation benchmarks show that soft-token fusion improves retrieval performance over LLM-based baselines, and that interaction-based fusion is more effective than direct concatenation of heterogeneous soft tokens.

Zhe Xu, Ankit Peshin, Chiyu Zhang et al. · arXiv.org · 0 citations

Preprint Aug 2026

Empowering Compact LLMs with Fusion of Layer-wise Exits for Recommendation

The Fusion of Layer-wise Exits for Sequential Recommendation (FLEXRec), a discriminative framework that enhances compact LLMs while retaining scalable full-corpus ranking and achieves state-of-the-art accuracy among competing methods while remaining highly efficient.

Xurong Liang, Tong Chen, Q. Nguyen et al. · 0 citations