Optimizing for Difference: LLM-Driven Benchmark Design for Maximum Solver Discriminability
TL;DR
This work proposes an automated framework that generates and refines benchmark suites using large language models (LLMs) and evolutionary search and shows that this combined approach substantially increases discriminability, improving scores from approximately 2.04 to 2.58.
Abstract
The GECCO Benchmark Design Challenge formulates benchmark construction as the problem of maximizing performance differences between optimizers, measured via pairwise rank distances. We approach this as a meta-optimization task and propose an automated framework that generates and refines benchmark suites using large language models (LLMs) and evolutionary search. Candidate functions are synthesized through LLM-driven evolution and evaluated directly on their ability to discriminate between five standard optimizers under a fixed evaluation budget. We further optimise the suite by combinatorial optimization of all generated functions over all considered dimensionalities in the range of [2,80). Our results show that this combined approach substantially increases discriminability, improving scores from approximately 2.04 to 2.58. These findings highlight that effective benchmark design requires not only diverse function structures but also careful control of problem dimensionality, and demonstrate the potential of learning-driven methods for constructing targeted benchmarking suites.