Data-Efficient Online Training for Direct Alignment in LLMs

Abstract

In recent years, online Direct Alignment from Preferences (DAP) has emerged as a popular alternative for Reinforcement Learning from Human Feedback (RLHF) due to its training stability and simplicity. In online DAP, training relies on preference data, each composed of a question and a pair of large language model (LLM) responses. However, annotating preference data, i.e., generating responses for questions, and using these data to train the RLHF model are computationally expensive. To address this, we propose DOTA, a data selection framework that minimizes the cost of generating preference data, while still ensuring the quality of training. First, we propose a theoretically grounded metric called Preference Perplexity (PFP) that enables us to design a low cost, gradient-based method to effectively estimate the contribution of each preference data point to model performance — critical to data selection. Second, rather than first generating responses for all candidate questions and then selecting preference data points by measuring their PFP, we design an iterative end-to-end framework that only has to generate responses for a small subset of questions, without missing valuable data points. Experiments on UltraChat-200k and HH-RLHF across 13 downstream tasks demonstrate that DOTA reduces computation cost by a factor of three on LLaMA-3-8B, Qwen-3-4B, and Qwen-3-1.7B, without compromising training effectiveness.