Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard
Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably...
Michael Hardy, Ruhana Azam, Anka Reuel et al.
· 0 citations