Skip to content

PowerBench: A Benchmark for Agentic Retrieval and Reasoning in Power Systems

Sep 2026 · arXiv.org · Vol abs/2609.34492 · 0 citations · 28 references
Computer Science

TL;DR

This work proposes PowerBench, a generation framework that derives interconnected heterogeneous operational data through a common dependency chain, and a synthetic dataset generated by this framework, comprising a generation framework that derives interconnected heterogeneous operational data through a common dependency chain and a synthetic dataset generated by this framework.

Abstract

Large language model (LLM) agents offer new opportunities for automated analysis in industry. However, rigorous evaluation of such agents-for example, within power system scenarios-remains hindered: real operational data are confidential, and existing public resources fail to fully capture the chained dependencies and heterogeneous evidence. To address this gap, we propose PowerBench, comprising (1) a generation framework that derives interconnected heterogeneous operational data through a common dependency chain, and (2) a synthetic dataset generated by this framework. The dataset covers 761 devices across 100 device types, with 13.35 million hourly telemetry records spanning two years and 24,939 operational documents. Building on this dataset, we construct 300 questions across three task families that evaluate frontier LLMs'ability to complete analysis tasks that require autonomous evidence retrieval and reasoning across interconnected and heterogeneous data under restricted tool calls and time budgets. Results demonstrate that the evaluated frontier LLMs remain challenged on these tasks: the best model reaches only 74.2% joint accuracy. Our trace analysis further reveals that model performance varies across evidence discovery, content retrieval, tool use, reasoning over evidence, and answer submission. These findings provide detailed insights for evaluating LLM agents and guiding their reliable deployment in industry. The framework, dataset, and benchmark tasks are available at https://github.com/open-compass/PowerBench.

View source

Similar papers

Conference Open access Sep 2026

A Review on Test-Time Scaling for Agentic Large Language Models

A novel RAIE taxonomy along four scaling dimensions is proposed, which optimizes the entire thought process through search algorithms and self-verification, and introduces a task-oriented guideline for choosing the best TTS strategy.

Jia-Yu An, Zheng Chen, Yong-Cheng Jing et al. · 0 citations

BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents

BabelFlow is introduced, a benchmark-general agentic workflow that adapts existing agent benchmarks to new languages by analyzing runtime dependencies, coordinating structure-preserving translation, and combining multi-layer verification with human review to preserve task and evaluation semantics.

Peng Kuang, Yu-Chun Fan, Jiang-Nan Li et al. · 1 citation

AnyAct: Universal Action for Self-Evolving Agents

AnyAct is a universal action layer that unifies available capabilities into a self-evolving action space, enabling agents to operate efficiently and reliably in large-scale, dynamic tool ecosystems and optimizes for a balance between task success rate and execution cost.

Ling-Rui Xu, Ya Jiang, Jia-Chang Zhang et al. · 0 citations
#artificial intelligence Preprint Aug 2026

RestoreBench: Can AI Agents Restore Power Flow Convergence?

A benchmark that evaluates diagnostic and resolving non-convergent power flow cases across multiple LLMs and three architectures, providing a reproducible foundation for developing agentic AI systems for power system planning and operation.

Riccardo Mansutti, Andrea Pomarico, Robert Jakob et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Towards Reliable AI Data Scientists: Data Agents with Workflow Harnesses

This paper examines Data Agents from a harness-centric perspective, introducing a taxonomy of Data Agents and associated data environments, and identifying four open reliability problems: inactive semantic calibration, missing clarification, missing experience transfer, and the missing verification-repair repository.

Hua-Chi Zhou, Yu-Jing Zhang, Jia-He Du et al. · 0 citations
#small language model Preprint Aug 2026

Apodex 1.1: Scaling Agentic Intelligence for Complex Work

Apodex 1.1 reaches the leading performance band despite using a substantially smaller model than many frontier systems, and the 35B-parameter Apodex 1.1 Mini further retains strong working capability in a locally deployable form.

B. An, B. Li, B. Wang et al. · 3 citations · ⚡1

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.