Zichao Liu

PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling

PluginEval is introduced, a benchmark constructed through a two-stage framework that systematically mitigates limitations of Large Language Models and evaluates five model families, including proprietary models and models with open weights, analyze their performance across difficulty levels and error categories, and validate the judge through agreement with human annotations.