Do More Agents Help? Evaluating Accuracy and Cost in LLM Multi-Agent Workflows

Abstract

Choosing whether to replace a single-agent controller requires evidence on both task performance and inference expenditure. We present BenchAgent, a shared evaluation framework, and compare six multi-agent workflow configurations against its single-agent reference, BenchAgent Core, across ten reasoning, coding, and tool-use benchmarks. All configurations use GPT-4.1 and common tool and scoring interfaces; Core executions share the same step limits. We measure accuracy and recorded token use over three runs on fixed task sets, with 3,302 instances per configuration per run. EvoAgent reaches 79.70% mean pooled accuracy versus 76.65% for Core and exceeds Core in all three runs; the other five MAS trail by 1.71-8.76 percentage points. LLM-Debate gains 9.67 points on HumanEval but loses 1.96 points in pooled accuracy while consuming 1.39 times the tokens of Core. ChatEval gains 22.25 points on IFEval but trails Core in pooled accuracy with 4.34 times its token consumption. The broad comparison and targeted analyses of retained responses provide evidence for workflow selection and identify answer preservation and resource allocation as directions for subsequent multi-agent optimization.