An experimental design and analysis framework for systematically evaluating this discovery process, quantifying its variability, and identifying important factors is proposed and demonstrated on a testbed of networked word-forming games with insightful findings on reasoning effort with respect to cost and process complexity.
Abstract
Large language model coding agents increasingly perform open-ended data modeling and analysis. These agents are stochastic and adaptive, and therefore their autonomous model discovery behavior cannot be adequately characterized by a single benchmark run. In this work, we propose an experimental design and analysis framework for systematically evaluating this discovery process, quantifying its variability, and identifying important factors. The proposed framework treats these agents as stochastic model-discovery operators, which map task-specific discovery data and an optimization target to a fitted model. Specifically, we investigate two such operators, Codex and Claude Code, under controlled experimental factors including agent's reasoning effort, task, optimization metric, and composition of training data. For each agent-task-metric combination, regression models and inference are conducted for multiple responses such as output quality, dollar cost, wall-clock time, and process complexity. Furthermore, we develop a utility-aligned canonical decomposition to characterize the dominant direction of the reasoning-effort effect and to assess whether that direction aligns with a performance-cost utility direction. The proposed framework is demonstrated on a testbed of networked word-forming games with insightful findings on reasoning effort with respect to cost and process complexity.
We investigate the use of large language models (LLMs) as evolutionary operators for optimizing the architecture of multi-agent systems (MAS) - a representative instance of LLM-driven optimization over attributed-graph artifacts that arise in generative design. Given a task description, an initial MAS configuration is generated and then refined on a per-task basis through two optimization strategies: (1) a population-based evolutionary approach where LLMs serve as mutation, crossover, and selection operators, with an LLM-as-a-Judge ensemble providing the surrogate fitness signal; and (2) a simpler iterative regeneration scheme that uses textual feedback from LLM judges to guide single-solution improvement without maintaining a population. We evaluate both approaches on the GAIA benchmark. Our experiments reveal a critical sensitivity to the judge model's quality: a capable judge (Claude 4 Sonnet) yields +14.3 and +6.67 percentage-point accuracy improvements for evolutionary and iterative approaches, respectively, whereas a weaker judge (Gemini-2.5-Flash) provides no benefit or even degrades performance below the unoptimized baseline. Both approaches incur substantial computational costs, raising questions about practical cost-effectiveness. These results highlight fundamental challenges of surrogate fitness functions in LLM-driven evolutionary optimization and offer practical insights for applying evolutionary methods to MAS architecture design.
V. Akhmerov, Dmitry Gilemkhanov, Jerzy Kamiński et al.· GECCO Companion· 0 citations
We study what happens when a single general-purpose large language model acts as the sole researcher on a long-horizon neural architecture design problem. The agent receives a scientific question, an initial hypothesis and motivation, a compute budget, and research affordances (source and experiment management, experiment tracking, literature access, and persistent memory), then autonomously proposes, implements, evaluates, and records experiments over an extended period. The study comprises three phases, separated by human-declared transitions, that progressively expand the agent's tool surface or problem scale. Across approximately 100 sequential experiments, the agent improves a non-standard Vision Transformer from a weak baseline to a stronger, efficient model on small benchmarks and a usable but sub-SOTA model on ImageNet-1K, while producing a dense behavioural trace. We report four findings.(i)Productivity exhibits a clear phase structure: rapid early gains, a multi-dozen-hypothesis saturation wall, and recovery, with recovery triggered by expanding the action surface rather than changing the underlying model.(ii)A single early hypothesis contributes more to accuracy gain, with later improvements long-tailed.(iii)The preference for greedy, incremental hypotheses is largely workflow-induced: a commit-or-discard evaluation rule is isomorphic to greedy hill-climbing; the remainder reflects risk aversion after bold failures and anchoring on familiar literature. (iv)The agent independently rediscovers established results and, in the unfamiliar regime of pure channel attention, overturns a standard design choice. We conclude that workflow design was at least as influential as agent capability in this study and propose diversified search, budgeted moonshot hypotheses, explicit forks, and regime-aware re-validation as testable directions for future autonomous research.
This work extends the classical Schelling segregation model with a hybrid population: ordinary agents classify neighbors using the standard symbolic rule, while one agent delegates this task to an LLM through tool calls, providing a minimal but controlled setting where the semantic, operational, and computational behavior of LLM-based decisions can be studied inside an otherwise standard ABM.
Stefano Blando, Emanuele Guerrazzi, R. Porcedda et al.· arXiv.org· 0 citations
CEDAR is an autonomous method that uses Large Language Model agents to discover complex systems satisfying user-specified behavioral goals, enabling goal-directed discovery of complex system behaviors while preserving solution diversity, and its LLM-based interpretability reveals how structural changes drive emergent behavior.
Experiments show that SciDisco-14B reaches state-of-the-art on hypothesis-driven scientific data analysis benchmarks, and SciDisco-14B reaches state-of-the-art on hypothesis-driven scientific data analysis benchmarks.
The MEDA system is introduced, an LLM- and SR-powered agentic framework for discovering ordinary-differential-equation models of biological and biologically inspired dynamical systems and shows that knowledge-guided formalization and mechanistic constraints are load-bearing components, whereas numerical fitting alone can preserve trajectory-compatible but biologically incorrect equations.
D. Krongauz, A. Zulti, Eran Segal et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
