Agent-driven scientific experimentation is emerging across domains such as chemistry, biology, and materials, yet each tool class imposes its own execution discipline. Network experimentation requires more than one-shot topology or configuration synthesis: an experimenter must plan a task, operate a live and evolving network, interpret feedback, refine intermediate state, and validate the resulting behavior. This poster presents a Network Experimentation Harness for emulator-backed network experiments, helping LLM agents operate across these stateful workflows. The Harness pairs a semantic action interface with reusable experimentation skills to handle sequencing, timing, and verification that a careful experimenter would perform by hand. A preliminary study on GNS3-based network protocol experiments shows that this approach reduces wall-clock time by 47% and 37%, and token use by 81% and 76%, on average versus raw GNS3 access and a Python wrapper (GNS3Fy), respectively.
Chenguang Du, Chang Liu, Lei Zhang et al.· Asia-Pacific Workshop on Net...· 0 citations
Large language model (LLM) agents are increasingly attractive for automating network configuration, yet their reliability and failure patterns are poorly understood. An essential prerequisite is to assess such agents in a realistic but risk-free environment. Existing benchmarks, however, fall short: they often treat configuration as static command generation or rely on overly simplified settings. Such evaluations understate the core challenges of network configuration, where correctness requires reasoning about protocol complexity and topology dependence. We present NetConfArena, an executable benchmark for evaluating LLM agents in closed-loop network configuration. NetConfArena places agents in emulated multi-device networks, provides a standardized and compact action interface for task execution, and evaluates the resulting network behavior with hidden task-specific executable test cases. The benchmark relies on an LLM-assisted, emulation-grounded pipeline, which converts human-oriented network materials into reusable parameterized task templates. We evaluate representative LLM agents on 480 task instances instantiated from 96 protocol-focused task templates, yielding 3840 execution trajectories, and show that failures are not limited to command errors. The failures also reveal gaps in task-specification adherence and robust planning and execution. These findings suggest two future directions: using validated trajectories as supervision signals to improve foundation models, and designing harness mechanisms that make agent execution more reliable and accountable.
Chang Liu, Xiaohui Xie, Xinyi Chen et al.· 0 citations