Evolving Multi-Agent Systems with Large Language Models: Surrogate Fitness Limitations and Empirical Insights
We investigate the use of large language models (LLMs) as evolutionary operators for optimizing the architecture of multi-agent systems (MAS) - a representative instance of LLM-driven optimization over attributed-graph artifacts that arise in generative design. Given a task description, an initial MAS configuration is generated and then refined on a per-task basis through two optimization strategies: (1) a population-based evolutionary approach where LLMs serve as mutation, crossover, and selection operators, with an LLM-as-a-Judge ensemble providing the surrogate fitness signal; and (2) a simpler iterative regeneration scheme that uses textual feedback from LLM judges to guide single-solution improvement without maintaining a population. We evaluate both approaches on the GAIA benchmark. Our experiments reveal a critical sensitivity to the judge model's quality: a capable judge (Claude 4 Sonnet) yields +14.3 and +6.67 percentage-point accuracy improvements for evolutionary and iterative approaches, respectively, whereas a weaker judge (Gemini-2.5-Flash) provides no benefit or even degrades performance below the unoptimized baseline. Both approaches incur substantial computational costs, raising questions about practical cost-effectiveness. These results highlight fundamental challenges of surrogate fitness functions in LLM-driven evolutionary optimization and offer practical insights for applying evolutionary methods to MAS architecture design.