Skip to content
Open access

Benchmarking Performance Judgment in Open-Weight LLM Controller Tuning: Control Knowledge Does Not Ensure Reliable Gain Ranking

Sep 2026 · Processes · Vol 14, pp. 2954 · 1 citation · 35 references

TL;DR

A simulator-grounded performance-judgment benchmark evaluates four open-weight checkpoints across a quadruple-tank process, a recycle reactor, and a synthetic 3-by-3 cyclic plant, finding that outputs are associated with lower gain magnitudes, but the association varies with scaling, plant, and controller sampling.

Abstract

Large language models (LLMs) are increasingly embedded in control-design workflows, yet their ability to compare candidate controllers remains uncertain. A simulator-grounded performance-judgment benchmark evaluates four open-weight checkpoints across a quadruple-tank process, a recycle reactor, and a synthetic 3-by-3 cyclic plant. Declarative control-theory accuracy is nearly saturated (98–100% pooled by plant), whereas qualitative-description accuracy for ranking PI gain sets is 46.8%, 31.8%, and 53.2%, respectively. Outputs are associated with lower gain magnitudes, but the association varies with scaling, plant, and controller sampling. Across 271 replay-stable pairs, IAE rankings agree with ISE rankings on 95.2%; under a combined simulator perturbation, 88.7% of seed–item labels are preserved. On fixed gain pairs, equations and 14-point response trajectories raise pooled accuracy only from 46.8% to 51.2% and 51.0%. All 12 prespecified trajectory-versus-qualitative intervals include zero, and none survives Holm correction. Swapping displayed trajectory values changes 9.5% of parsed choices. By contrast, a numerical scaffold derived from the same samples raises evidence-consistent accuracy from 51.1% to 88.1%, with 11 of 12 contrasts surviving Holm correction. The main bottleneck under these prompts is extracting and aggregating raw numerical evidence, not the final comparison alone. This result does not identify a unique mechanism or establish a general absence of dynamic reasoning. Simulator-based validation remains necessary before LLM judgments are used for controller tuning.

Read PDF

Similar papers

#large language models Open access Oct 2026

Benchmarking the responsiveness of LLM-based agentic systems under controlled mid-task perturbations

A responsiveness evaluation framework that pairs three agent architectures (ReAct, Reflexion, and Plan-and-Execute) with three deterministic synthetic environments (tool planning, debugging, and constraint planning) and injects tool denial, goal modification, or constraint injection at graded severity is presented.

Ayush Balyan, Surbhi Sharma · 0 citations
Preprint Nov 2026

TuneBench: Benchmarking AI Agents for Power System Dynamic Model Calibration

TuneBench is introduced, a benchmark for evaluating Agentic AI in power system dynamic model calibration, where controller parameters are adjusted to reproduce target dynamic responses in LLM-based agents.

Andrea Pomarico, S. Tessitore, G. Giannuzzi et al. · 0 citations
Preprint Nov 2026

NCP: Verifier-Guided Optimization of Weight-Space Control Policies

We present the Neural Control Plane (NCP), a closed-loop system that observes language-model execution, synthesizes candidate weight-space intervention rules, evaluates them, and maintains a versioned table of admitted policies. NCP separates a pretrained model's inference data plane from the control policy that select...

Ekrem Aksoy · 1 citation
Preprint Oct 2026

RealControlBench: A Benchmark for Interpretability-Informed Fine-Grained Control of LLMs and Agents

The evaluation shows that existing steering methods remain unreliable across realistic control objectives: they often fail to consistently induce the intended behavior and can substantially degrade the model's underlying task performance, suggesting that strong steering effects in controlled settings do not necessarily...

Xin-Yi Cai, Bin Hu, Ming-Yu Jin et al. · 0 citations
Preprint Oct 2026

Harnessing LLMs with Direction-Scope Control for Combinatorial Solver Refinement

Harnesses are increasingly unlocking the potential of large language model (LLM) agents by making their actions more controllable. In automatic heuristic design (AHD) for combinatorial optimization (CO), LLMs iteratively revise solvers using execution and verification feedback. However, these revisions often break prev...

Yi-Lin Cao, An-Qi Li, Hao-Ran Li et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.