Dynamically Allocating Evaluation Effort for Model Ranking
This work formalizes multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms, where pulling an arm corresponds to human-evaluating a model, and proves the optimality of the proposed algorithms and shows that it improves discrimination between top-performing models.
Vilém Zouhar, Julia Kreutzer, A. Lavie et al.
· 0 citations