Skip to content

Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment

Oct 2026 · 0 citations
Computer Science

TL;DR

Welfare-Aware Preference Acquisition (WAPA) uses auxiliary reference models to estimate query gains and individual utilities, then selects queries by estimated marginal welfare gain while updating recipient utilities, and it is proved exact optimality of the acquisition algorithm for the estimated welfare objective under a fixed query budget and model predictions.

Abstract

Pluralistic alignment needs to accommodate diverse preferences under limited human feedback. Using AI simulators to label preference comparisons reduces annotation costs, but variation in simulation accuracy across individuals can leave some people's preferences poorly represented in training. We study how to allocate human feedback across people to construct training data for pluralistic alignment, and how this allocation affects the trained policy. To make this allocation explicit and controllable, we define a social welfare objective over individual supervision utilities that measure label quality. Each query's marginal welfare gain depends on its expected supervision improvement, the recipient's current utility, and application-designated priority. Welfare-Aware Preference Acquisition (WAPA) uses auxiliary reference models to estimate query gains and individual utilities, then selects queries by estimated marginal welfare gain while updating recipient utilities. We prove exact optimality of the acquisition algorithm for the estimated welfare objective under a fixed query budget and model predictions. Experiments on six Roleplay collections and four OpinionQA regions demonstrate improved supervision and controllable allocation tradeoffs. Policy training on four Roleplay collections improves preference prediction for both seen and unseen individuals. When prioritizing a designated group and poorly represented individuals, acquiring human labels for 20% of comparisons recovers 75-86% of the average predictive utility gap between policies trained entirely on AI-generated labels and entirely on ground-truth labels.

View source

Similar papers

Preprint Nov 2026

Objective-Conditioned Active Preference Learning for Multi-Objective Alignment

Aligning large language models (LLMs) with several objectives, such as helpfulness, harmlessness, and honesty, typically relies on preference judgment for each objective, growing the annotation cost linearly with the number of objectives. Active preference learning can reduce this cost by selecting the comparisons whos...

Duc Anh Vu, T. Nguyen, Hoang Pham et al. · 0 citations

Optimal Design for Active Preference Learning with Biased LLM Judges

Nuisance-Adjusted Optimal Design is proposed, a comparison-selection strategy that prioritizes policy-relevant target information after nuisance adjustment and uses the Frank-Wolfe algorithm for optimization and establishes a sharp conditional local asymptotic minimax lower bound on policy risk and construct an estimat...

Zhong-Man Du, Hui-Ming Zhang, Hao-Dong Zhu et al. · 1 citation
Open access Oct 2026

Multi-Evaluator Preference Alignment Under Scarce Human Feedback Budgets: Theoretical Frameworks and Aggregation Strategies

The alignment of generative artificial intelligence systems with human values and intentions relies heavily on iterative feedback mechanisms. As models scale in complexity, the demand for high-quality human annotation escalates, presenting a critical bottleneck when financial and temporal budgets are constrained. This...

Timothy Pak-Ho Lai, David Kan · 0 citations
#natural language process... Preprint Oct 2026

Adaptive Personalization Under Preference Epistemic Uncertainty

Human preferences change over time: an established preference may remain valid over time, become uncertain as context shifts, or be explicitly revised. Effective personalization requires conversational AI to adapt its response strategy to evolving preferences rather than relying on passive memory retrieval. We formaliz...

Priyanka Dey, Raja Das, Vikas Bahirwani et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Can You Trust Your Preference Models? Stackelberg Games for Alignment

Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train policies to search for comparisons on which an evaluator may reverse the underlying preference, providing candidates for additional supervision. Since human preferences are un...

R. Formenti, Aditya Agrawal, Vikas K. Garg · 0 citations
#machine learning Preprint Oct 2026

Groupwise Distortion Guarantees for Preference-Based Alignment

An efficient algorithm is given, GLHF, that learns a single group-conditioned policy from one comparison per user, and substantially reduces worst-group distortion relative to NLHF and other group-agnostic baselines.

Jacob Brodkey, Roberto Tamez, Aaron Roth · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.