Welfare-Aware Preference Acquisition (WAPA) uses auxiliary reference models to estimate query gains and individual utilities, then selects queries by estimated marginal welfare gain while updating recipient utilities, and it is proved exact optimality of the acquisition algorithm for the estimated welfare objective under a fixed query budget and model predictions.
Abstract
Pluralistic alignment needs to accommodate diverse preferences under limited human feedback. Using AI simulators to label preference comparisons reduces annotation costs, but variation in simulation accuracy across individuals can leave some people's preferences poorly represented in training. We study how to allocate human feedback across people to construct training data for pluralistic alignment, and how this allocation affects the trained policy. To make this allocation explicit and controllable, we define a social welfare objective over individual supervision utilities that measure label quality. Each query's marginal welfare gain depends on its expected supervision improvement, the recipient's current utility, and application-designated priority. Welfare-Aware Preference Acquisition (WAPA) uses auxiliary reference models to estimate query gains and individual utilities, then selects queries by estimated marginal welfare gain while updating recipient utilities. We prove exact optimality of the acquisition algorithm for the estimated welfare objective under a fixed query budget and model predictions. Experiments on six Roleplay collections and four OpinionQA regions demonstrate improved supervision and controllable allocation tradeoffs. Policy training on four Roleplay collections improves preference prediction for both seen and unseen individuals. When prioritizing a designated group and poorly represented individuals, acquiring human labels for 20% of comparisons recovers 75-86% of the average predictive utility gap between policies trained entirely on AI-generated labels and entirely on ground-truth labels.
Aligning large language models (LLMs) with several objectives, such as helpfulness, harmlessness, and honesty, typically relies on preference judgment for each objective, growing the annotation cost linearly with the number of objectives. Active preference learning can reduce this cost by selecting the comparisons whos...
Duc Anh Vu, T. Nguyen, Hoang Pham et al.· 0 citations
Nuisance-Adjusted Optimal Design is proposed, a comparison-selection strategy that prioritizes policy-relevant target information after nuisance adjustment and uses the Frank-Wolfe algorithm for optimization and establishes a sharp conditional local asymptotic minimax lower bound on policy risk and construct an estimat...
Zhong-Man Du, Hui-Ming Zhang, Hao-Dong Zhu et al.· arXiv.org· 1 citation
The alignment of generative artificial intelligence systems with human values and intentions relies heavily on iterative feedback mechanisms. As models scale in complexity, the demand for high-quality human annotation escalates, presenting a critical bottleneck when financial and temporal budgets are constrained. This...
Timothy Pak-Ho Lai, David Kan· INTERNATIONAL JOURNAL OF RES...· 0 citations
Human preferences change over time: an established preference may remain valid over time, become uncertain as context shifts, or be explicitly revised. Effective personalization requires conversational AI to adapt its response strategy to evolving preferences rather than relying on passive memory retrieval. We formaliz...
Priyanka Dey, Raja Das, Vikas Bahirwani et al.· 0 citations
Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train policies to search for comparisons on which an evaluator may reverse the underlying preference, providing candidates for additional supervision. Since human preferences are un...
R. Formenti, Aditya Agrawal, Vikas K. Garg· 0 citations
An efficient algorithm is given, GLHF, that learns a single group-conditioned policy from one comparison per user, and substantially reduces worst-group distortion relative to NLHF and other group-agnostic baselines.
Jacob Brodkey, Roberto Tamez, Aaron Roth· 0 citations
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.