We consider the problem of learning a mixture of $k$ Plackett-Luce models given multi-way ranking responses from annotators that may represent heterogeneous underlying preferences. This problem has many applications in AI alignment and preference optimization. Prior work has studied mixtures of Bradley-Terry models from pairwise comparisons. However, uncovering mixture models is theoretically unidentifiable when $k$ exceeds $m/2$, where $m$ is the length of a ranking. We propose an efficient implementation to address this limitation, which involves first augmenting the rankings to a larger size by generating new responses from a base language model, followed by a gradient-based estimation to reduce inference cost in the input embedding space. Based on this procedure, we then design an expectation-maximization algorithm with these two steps to fit a mixture of Plackett-Luce models, called MoPLEx. Extensive experiments are conducted to verify this approach. First, we show that the gradient-based approximation estimates true probabilities with less than 5% error on models with up to 34 billion parameters. Second, we show that MoPLEx improves clustering and ranking accuracy by an average of 43.7% and 15.2% over baselines using single ranking and mixtures of Bradley-Terry models, on preference optimization datasets. These results demonstrate the effectiveness of MoPLEx for tackling multi-way rankings from heterogeneous preferences through measuring alignment between gradients.
In recent years, online Direct Alignment from Preferences (DAP) has emerged as a popular alternative for Reinforcement Learning from Human Feedback (RLHF) due to its training stability and simplicity. In online DAP, training relies on preference data, each composed of a question and a pair of large language model (LLM) responses. However, annotating preference data, i.e., generating responses for questions, and using these data to train the RLHF model are computationally expensive. To address this, we propose DOTA, a data selection framework that minimizes the cost of generating preference data, while still ensuring the quality of training. First, we propose a theoretically grounded metric called Preference Perplexity (PFP) that enables us to design a low cost, gradient-based method to effectively estimate the contribution of each preference data point to model performance — critical to data selection. Second, rather than first generating responses for all candidate questions and then selecting preference data points by measuring their PFP, we design an iterative end-to-end framework that only has to generate responses for a small subset of questions, without missing valuable data points. Experiments on UltraChat-200k and HH-RLHF across 13 downstream tasks demonstrate that DOTA reduces computation cost by a factor of three on LLaMA-3-8B, Qwen-3-4B, and Qwen-3-1.7B, without compromising training effectiveness.
Unknown authors· Proceedings of the VLDB Endo...· 0 citations
Experiments show that DIAG boosts yield across iterations and delivers stronger reasoning performance under an iso-effective training budget, demonstrating that it can distill more informative preference supervision for mathematical reasoning.
Guhan Chen, Songtao Tian, Bo-Han Li et al.· 0 citations
Multi-objective ranking serves as the backbone of industrial information retrieval, requiring a holistic assessment of documents across dimensions such as Relevance, Authority, and Recency. The prevailing industry paradigm relies on ensembles of specialized BERT-based models, which are costly to maintain and fundamentally limited in expressiveness by the modest model size of discriminative encoders. While Large Language Models (LLMs) offer a path to unified reasoning, their auto-regressive latency is prohibitive for real-time serving. In this paper, we propose UniRank, a Homogeneous Scaling framework that consolidates multiple ranking objectives into a single, unified generative model. To reconcile the conflict between deep reasoning and inference speed, we introduce two paradigmatic innovations: (1) Parallel Prediction via Task-Specific Anchors: We replace slow sequential decoding with a parallel mechanism that predicts all objective scores simultaneously in a single forward pass, achieving O(1) inference complexity. (2) Reinforcement Learning Optimization with Posterior CoT: We propose a novel training framework that synergizes Posterior Chain-of-Thought with GRPO. By forcing the model to generate rationales after scoring and aligning them via policy optimization, we effectively distill complex reasoning capabilities into the anchor tokens without incurring inference-time overhead. Extensive experiments on the MS MARCO benchmark and a massive industrial dataset from Tencent demonstrate that UniRank significantly outperforms BERT-Large and matches the performance of larger LLMs, all while maintaining strictly low latency. Online A/B testing further confirms its practical value, showing that UniRank simplifies the deployment pipeline while delivering a significant lift in core business metrics.
Dezhi Ye, Junwei Hu, Xiaoyang Chen et al.· Proceedings of the 32nd ACM...· 0 citations
Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling model capacity while preserving efficient inference in large foundation models. However, most MoE models use a fixed top-$k$ expert selection policy, assigning the same expert budget to every token even when fewer experts may be sufficient. Inference-time dynamic top-$k$ routing can reduce computation without retraining, but existing methods often overlook the distributional shift caused by deviating from the training-time routing configuration. We show that reducing the number of activated experts consistently increases the RMS scale and variance of SMoE outputs, inducing a representation mismatch that contributes to downstream performance degradation in addition to the loss of expert capacity. To address this correctable component, we propose Layer-wise Distribution Alignment (LDA), a lightweight inference-time correction that uses layer-wise calibration statistics to align reduced-routing representations with the default configuration. Across multiple SMoE LLMs, benchmarks, and routing strategies, LDA recovers much of the performance lost induced by the distributional shift under reduced routing while preserving sparse-inference efficiency with negligible overhead.
Dohyeon Kim, Bedionita Soro, Sung Ju Hwang· 0 citations
Aligning a language agent to several objectives at once is a persistent failure mode of preference-based training: when objectives are combined additively, optimization collapses onto whichever is cheapest to improve and sacrifices the rest, so a support agent learns to sound warm while giving no real help. The root issue is that an additive reward has no notion of balance. We introduce Mint (MIN-selection preference disTillation), a one-line change to preference distillation: rather than ranking sampled candidates by a weighted sum of rewards, we rank them by their weakest objective, distilling the best-balanced candidate over the most lopsided one with an unchanged DPO objective. This is the p ->negative infinity limit of a generalized-mean family spanning additive to worst-case selection. Across cooperative emotional support and adversarial negotiation, min-selection lifts both objectives while sharply cutting their imbalance; on emotional support it raises the weaker axis from 0.37 to 0.64 (p<10^-40), surpassing human experts and persisting across full multi-turn rollouts. A turn-by-turn analysis yields our central finding: min-selection corrects imbalance in proportion to how imbalanced the reference policy is, and its benefit endures over an interaction precisely as long as that imbalance does.
Tony Tu, Sayan Chakraborty, Ruomeng Xu et al.· 0 citations
We introduce a Bayesian latent block model that jointly partitions assessors and items under a Plackett--Luce observation model. Assessors are assigned to $C$ clusters and items to $K$ blocks; items in a block share a common strength parameter within each assessor cluster, yielding a parsimonious $C\times K$ co-clustering representation. Independent Gnedin priors infer $C$ and $K$. Data augmentation gives conjugate Gibbs updates and a tractable MCMC sampler with split-merge moves. Simulations characterize recovery and posterior uncertainty as signal, ranking depth, and group balance vary. Applied to the cancer gene atlas (TCGA) pan-cancer top-500 gene-expression rankings, the model reveals tissue-driven sample structure while compressing gene-level heterogeneity into interpretable blocks. Rank-based GSEA of posterior gene scores supports biological interpretation.
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
