This work proposes PowerBench, a generation framework that derives interconnected heterogeneous operational data through a common dependency chain, and a synthetic dataset generated by this framework, comprising a generation framework that derives interconnected heterogeneous operational data through a common dependency chain and a synthetic dataset generated by this framework.
Abstract
Large language model (LLM) agents offer new opportunities for automated analysis in industry. However, rigorous evaluation of such agents-for example, within power system scenarios-remains hindered: real operational data are confidential, and existing public resources fail to fully capture the chained dependencies and heterogeneous evidence. To address this gap, we propose PowerBench, comprising (1) a generation framework that derives interconnected heterogeneous operational data through a common dependency chain, and (2) a synthetic dataset generated by this framework. The dataset covers 761 devices across 100 device types, with 13.35 million hourly telemetry records spanning two years and 24,939 operational documents. Building on this dataset, we construct 300 questions across three task families that evaluate frontier LLMs'ability to complete analysis tasks that require autonomous evidence retrieval and reasoning across interconnected and heterogeneous data under restricted tool calls and time budgets. Results demonstrate that the evaluated frontier LLMs remain challenged on these tasks: the best model reaches only 74.2% joint accuracy. Our trace analysis further reveals that model performance varies across evidence discovery, content retrieval, tool use, reasoning over evidence, and answer submission. These findings provide detailed insights for evaluating LLM agents and guiding their reliable deployment in industry. The framework, dataset, and benchmark tasks are available at https://github.com/open-compass/PowerBench.
A novel RAIE taxonomy along four scaling dimensions is proposed, which optimizes the entire thought process through search algorithms and self-verification, and introduces a task-oriented guideline for choosing the best TTS strategy.
Jia-Yu An, Zheng Chen, Yong-Cheng Jing et al.· Proceedings of the Thirty-Fi...· 0 citations
BabelFlow is introduced, a benchmark-general agentic workflow that adapts existing agent benchmarks to new languages by analyzing runtime dependencies, coordinating structure-preserving translation, and combining multi-layer verification with human review to preserve task and evaluation semantics.
Peng Kuang, Yu-Chun Fan, Jiang-Nan Li et al.· arXiv.org· 1 citation
AnyAct is a universal action layer that unifies available capabilities into a self-evolving action space, enabling agents to operate efficiently and reliably in large-scale, dynamic tool ecosystems and optimizes for a balance between task success rate and execution cost.
Ling-Rui Xu, Ya Jiang, Jia-Chang Zhang et al.· arXiv.org· 0 citations
A benchmark that evaluates diagnostic and resolving non-convergent power flow cases across multiple LLMs and three architectures, providing a reproducible foundation for developing agentic AI systems for power system planning and operation.
Riccardo Mansutti, Andrea Pomarico, Robert Jakob et al.· 0 citations
This paper examines Data Agents from a harness-centric perspective, introducing a taxonomy of Data Agents and associated data environments, and identifying four open reliability problems: inactive semantic calibration, missing clarification, missing experience transfer, and the missing verification-repair repository.
Hua-Chi Zhou, Yu-Jing Zhang, Jia-He Du et al.· 0 citations
Apodex 1.1 reaches the leading performance band despite using a substantially smaller model than many frontier systems, and the 35B-parameter Apodex 1.1 Mini further retains strong working capability in a locally deployable form.