Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization
It is shown that, when a model is aligned sequentially source by source, exact optimization makes each stage contribute additively to the log-probability of a full response, up to a prompt-dependent normalization constant, reducing the alignment cost of Shapley-based valuation from exponential to linear.
Mélissa Tamine, Otmane Sakhi, Benjamin Heymann et al.
· 1 citation