The end-to-end treatment policy delivered a statistically significant $+7.20\% lift in the primary long-term-value metric, demonstrating the feasibility of production-scale causal optimization under business constraints.
Abstract
Large-scale targeting and recommendation systems are typically built around predictive scores fed into heuristic or local allocation. When the business goal is incremental impact, as in marketing campaigns, incentives, and notifications, this paradigm systematically misallocates resources toward users who would have acted anyway. We present a decision-centric framework that instead optimizes causal effects under global constraints, aligning three components under a single objective: a causal neural network with a Transformer backbone for individual treatment-effect estimation, a Bayesian neural-bandit layer for uncertainty-aware exploration, and a dual-based large-scale linear-programming layer for constrained allocation. The framework also supports sequential context and multi-outcome, attribute-conditioned scoring through a Transformer encoder and outcome embeddings. We evaluate it with offline simulations on a public bandit dataset, targeted architectural ablations, and an online A/B test on LinkedIn Feed marketing traffic. We also distill production lessons on causal training-data construction and cost and delivery control, which were critical to successful deployment. The end-to-end treatment policy delivered a statistically significant $+7.20\%$ lift in the primary long-term-value metric, demonstrating the feasibility of production-scale causal optimization under business constraints.
Recommendation impressions are a finite resource, hence delivering a recommendation to a user who would discover the content organically yields no incremental value and displaces other recommendations that could. We address this by extending an existing production recommendation model to a causal architecture using holdback data that is already collected as part of routine experimentation infrastructure, requiring no new data collection. A central challenge is that attribution windows differ between treated and holdback observations: treated users are attributed a stream within a short direct-response window, while holdback users are attributed organic streams over a multi-day window. This mismatch makes naive treatment-effect subtraction invalid. We resolve this with a dual-threshold targeting policy that delivers a recommendation only when the probability of a treated stream is high and the probability of organic stream is low. In a production-scale A/B test on millions of Spotify users, this policy reduces recommendation impressions by 7% with no statistically significant reduction in overall recommended content consumption. We further show that joint training with holdback data improves calibration of the treated head relative to the production baseline, and argue this can be taken as evidence that causal models learn more generalisable representations than models trained on observational data alone.
Athanasios Vlontzos, David Gustafsson, Michael O'Riordan et al.· 0 citations
Educational platforms increasingly personalize which AI learning-path features (adaptive homework, learner choice) each student receives. The natural correlational baseline ranks students by predicted performance—deliver the feature to those expected to do well—a heuristic that need not identify who actually benefits. We formalize feature recommendation as a causal targeting-policy problem: rank students by the estimated conditional average treatment effect (CATE) of a feature and recommend to the top of the ranking. We prove three results: (i) causal top-CATE targeting maximizes policy value at any budget and weakly dominates predictive (outcome-based) targeting, strictly when the two rankings disagree; (ii) a split-sample doubly robust evaluation of targeting quality is leakage-free (null-exact in finite samples), whereas the naive in-sample version is optimistically biased; and (iii) greedily targeting by CATE traces the optimal cost–benefit (Qini) frontier, with the deployment rule “recommend when τ^>0.” We validate the method on 17 randomized embedded experiments from the ASSISTments platform. Because the 40 held-out splits re-partition the same students, we do not treat them as independent replicates: we calibrate every headline comparison against a within-experiment permutation null and an experiment-clustered bootstrap. Under that calibrated inference, causal targeting outperforms predictive targeting for adaptive homework at every budget (permutation p≤0.005, the resolution floor of 200 replicates; Holm-corrected p≤0.040), while for learner choice the same contrast is directionally consistent but not statistically significant (permutation p=0.23–0.38; clustered p=0.42). The direction is stable in both families: no leave-one-experiment-out refit reverses its sign. Predictive targeting is nonetheless the one rule that is reliably worse than the alternatives, because it recommends the feature to high-performing students who benefit least—realized benefit falls monotonically across predicted-performance deciles (from +0.087 in the lowest to −0.030 in the highest). Against a fuller baseline suite, causal (CATE) targeting does not beat random, a simple risk-based rule (target low performers), or treating everyone. Indeed, the estimated benefit ranking is close to noise—split-half rank agreement is ρ≈0.002–0.008 and its calibration slope is 0.018, far below the ideal of 1—so the gain over predictive targeting comes from avoiding an actively harmful ordering rather than from recovering individual benefit. A fairness analysis shows why this matters: predictive targeting is regressive, concentrating feature access on high-ability students, whereas causal and risk-based targeting reverse that gradient in this corpus; no policy differentiates by neighborhood opportunity zone. Group-conditional policy values, however, are not individually distinguishable from zero once dependence across students and experiments is accounted for; what survives resampling is the allocation itself—predictive targeting directs 0.33 fewer of its recommendations to low-ability than to high-ability students (95% CI [−0.46,−0.01], experiment-clustered)—so we frame the fairness result as improved access, not established equity gains. The actionable finding is therefore narrow and specific: outcome-based targeting systematically mis-allocates learning-path features and should be replaced by some benefit-aware rule; whether that rule needs to be a learned CATE model, rather than a simple risk-based heuristic, is not established by this corpus.
Yan-Fen Huang, Lin Wang, Weihua Bai et al.· Electronics· 0 citations
Modern marketing increasingly requires managers to deploy new content at scale, often with limited opportunity for prior testing. As a result, decisions about what to launch become strategic managerial choices under uncertainty rather than purely creative exercises. While generative AI makes the creation of new content fast and highly scalable, it simultaneously expands the set of options managers must evaluate, making reliable content selection increasingly difficult. We develop a framework for causal prediction that enables managers to evaluate and deploy novel marketing content generated by AI. The framework uses pretrained large language models to represent previously deployed content and learn how its features causally relate to outcomes. Using a rejection-sampling procedure, the framework screens new content proposed by generative AI to avoid extrapolation beyond what historical data can reliably support. In a large-scale email marketing application (3.3 million observations across 34 campaigns), the framework improves out-of-sample prediction and real-world deployment performance relative to standard approaches, enabling outcome-guided generation of higher-performing AI-generated content. The framework establishes a threshold based on how closely new content resembles past campaigns, separating cases where causal prediction is reliable from cases where direct experimentation is warranted. The framework has important implications for marketing decision making in a rapidly evolving environment where generative AI is transforming content creation and deployment.
Paul B. Ellickson, Wreetabrata Kar, James C. Reeder et al.· Journal of Marketing Researc...· 1 citation
Large language models (LLMs) can summarize heterogeneous user evidence in natural language, but current LLM recommenders often collapse enduring preferences, transient intent, and exposure-induced behavior into one profile. This makes recommendation vulnerable to feedback loops: repeated exposure is mistaken for preference, immediate clicks dominate delayed satisfaction, and fluent explanations need not reflect the ranking decision. We propose our method, a model-agnostic framework for long-horizon recommendation. Our method uses a frozen multimodal language model to convert item content and feedback into evidence-grounded semantic atoms, then maintains separate short-term, long-term, and exposure memories. Propensity-weighted updates reduce policy-induced exposure bias, while a conservative offline critic reranks candidates for delayed satisfaction under a behavior-support constraint. Explanations use only influential evidence atoms and are checked by counterfactual deletion. We provide an identification result and evaluate the framework in e-commerce-like, news-like, and short-video-like environments. Across ten seeds, our method improves discounted long-term value over the strongest alternative by 6.1%, 7.6%, and 6.7%, respectively. Twenty-seed paired ablations show significant value drops after removing propensity correction (0.739 +/- 0.191) or conservative support regularization (0.523 +/- 0.234). A frozen instruction language model also more than doubles semantic-atom NDCG over TF-IDF on a held-out paraphrase benchmark.
Foundation model(FM) for recommendation has shown strong ability to model long-horizon sequential user behavior. In practice, a single pretrained foundation model is often adapted to diverse downstream serving surfaces through Supervised Fine-Tuning(SFT). However, optimizing task-specific objectives such as clicks or likes does not necessarily align the serving policy with the business metrics that determine recommendation quality. We propose a three-phase progressive post-training framework that explicitly separates downstream adaptation from business-metric alignment. The adaptation stage is decomposed into Linear Probing(LP) and Full Fine-Tuning(FFT): LP first stabilizes randomly initialized downstream heads within a frozen pretrained representation space, and FFT then jointly specializes the full model for the target task. On top of this stabilized policy, Reinforcement Fine-Tuning(RFT) aligns the model with practical business objectives using a learned reward model. Rather than directly optimizing the serving policy on sparse business targets, we train the policy on dense implicit feedback and use business-metric supervision only for reward modeling. Offline experiments show that the progressive LP-FFT-RFT framework outperforms single-phase alternatives, and that reward-based alignment yields a stronger serving policy than directly using the reward model itself for ranking. Large-scale online A/B tests further show that the proposed framework improves production recommendation quality over a conventional non-foundation baseline. A reference implementation is available at https://github.com/webtoon/rec-fm-progressive-alignment
Oseong Choi, Hoe-Gang Kim, Jihoon Lee et al.· 0 citations
Advertising recommendation requires continuously tuning complex system parameters while balancing commercial returns and user experience. Recent work has introduced large language models (LLMs) with skill documents to assist this labor-intensive process, but skill optimization remains largely prompt-driven, lacking a principled mechanism to attribute rewards to specific document edits. To address this limitation, we propose Document-Mediated Reinforcement Learning (DMRL), a skill self-evolution framework that models skill document optimization as a sequence of structured editing actions. In DMRL, an upper-level agent performs controlled document edits, while a frozen lower-level task agent evaluates their effects through A/B testing. To address credit assignment and long-term outcomes, we introduce two key components: (1) Dual-Relative Policy Optimization (DRPO), a post-training policy optimization method for robust and risk-aware advantage estimation; and (2) Long-term Reward Predictor (LRP), which estimates long-term outcomes by modeling population heterogeneity with disentangled representation learning and cross-attention transfer. DMRL was deployed on a large-scale short-video ads platform and extensive empirical evaluation shows that DMRL outperforms state-of-the-art baselines across key advertising metrics
Wei Zhang, Hong-Ji Li, Song Sun et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
