This work formalizes multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms, where pulling an arm corresponds to human-evaluating a model, and proves the optimality of the proposed algorithms and shows that it improves discrimination between top-performing models.
Abstract
While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability. When identifying top-performing models, typical evaluation protocols waste effort by exhaustively evaluating all models on the entire benchmark, a safe but inefficient approach. In this work, we formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms, where pulling an arm corresponds to human-evaluating a model. By sampling adaptively based on the intermediate model rankings obtained on the samples so far, we can focus the annotation budget on the most competitive models. We prove the optimality of the proposed algorithms and show that it improves discrimination between top-performing models. This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.
Evolutionary Soups, a mixture-of-experts framework for fine-grained generation control, with gating networks trained via an evolutionary algorithm, achieves the best hypervolume, linear utility, and Tchebyshev utility among controllable methods on all tasks.
Lingxiao Kong, Steffen Staab, Cong Yang et al.· 0 citations
BudgetDoc is introduced, the first multimodal benchmark providing explicit supervision for model-budget-performance trade-offs across three document tasks, and DRB (Document-Reasoning Balancer), an approx.
Standard evaluation of large language models is challenged by varying the token generation budget, i.e., the maximum tokens a model may produce, across seven levels, evaluating four models on three reasoning benchmarks, and finding three findings that argue for budget-conditioned evaluation protocols.
Rodrigo Guedes de Souza, Alison R. Panisson· 1 citation
A consensus-based evaluation framework that measures relative preference among model-generated responses rather than absolute correctness rather than absolute correctness is introduced, offering an alternative perspective on response quality in scenarios where multiple valid answers exist.
Multi-objective ranking serves as the backbone of industrial information retrieval, requiring a holistic assessment of documents across dimensions such as Relevance, Authority, and Recency. The prevailing industry paradigm relies on ensembles of specialized BERT-based models, which are costly to maintain and fundamentally limited in expressiveness by the modest model size of discriminative encoders. While Large Language Models (LLMs) offer a path to unified reasoning, their auto-regressive latency is prohibitive for real-time serving. In this paper, we propose UniRank, a Homogeneous Scaling framework that consolidates multiple ranking objectives into a single, unified generative model. To reconcile the conflict between deep reasoning and inference speed, we introduce two paradigmatic innovations: (1) Parallel Prediction via Task-Specific Anchors: We replace slow sequential decoding with a parallel mechanism that predicts all objective scores simultaneously in a single forward pass, achieving O(1) inference complexity. (2) Reinforcement Learning Optimization with Posterior CoT: We propose a novel training framework that synergizes Posterior Chain-of-Thought with GRPO. By forcing the model to generate rationales after scoring and aligning them via policy optimization, we effectively distill complex reasoning capabilities into the anchor tokens without incurring inference-time overhead. Extensive experiments on the MS MARCO benchmark and a massive industrial dataset from Tencent demonstrate that UniRank significantly outperforms BERT-Large and matches the performance of larger LLMs, all while maintaining strictly low latency. Online A/B testing further confirms its practical value, showing that UniRank simplifies the deployment pipeline while delivering a significant lift in core business metrics.
Dezhi Ye, Junwei Hu, Xiaoyang Chen et al.· Proceedings of the 32nd ACM...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
