Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation
This work proposes prediction-powered evaluation, a framework that combines limited human judgments with large-scale automatic scores to obtain data-efficient system comparisons that are provably unbiased, and introduces the Prediction-Powered Saving Ratio (PPSR), a meta-metric that measures how much human annotation an automatic metric can save when used within prediction-powered evaluation.
Mingqi Gao, Anthony B. Sicilia, Weiye Shi
· 0 citations