Benchmarking the responsiveness of LLM-based agentic systems under controlled mid-task perturbations
TL;DR
A responsiveness evaluation framework that pairs three agent architectures (ReAct, Reflexion, and Plan-and-Execute) with three deterministic synthetic environments (tool planning, debugging, and constraint planning) and injects tool denial, goal modification, or constraint injection at graded severity is presented.
Abstract
Large language model (LLM) agents are increasingly deployed where tools fail, objectives change, and constraints tighten while a task is in progress. Prevailing benchmarks score only static end-task accuracy and therefore miss mid-episode responsiveness, which is the ability to adapt and recover after a controlled change. We present a responsiveness evaluation framework that pairs three agent architectures (ReAct, Reflexion, and Plan-and-Execute) with three deterministic synthetic environments (tool planning, debugging, and constraint planning) and injects tool denial, goal modification, or constraint injection at graded severity. Across two matched factorial grids of 4,320 episodes each—one with an explicit perturbation notice and one silent. ReAct attains the highest pooled reward (0.544), yet Plan-and-Execute leads on tool planning (0.711 vs. 0.532) while collapsing on constraint planning (0.114). Tool denial uniquely depresses behavioural recovery, and severity within [0.1, 0.9] yields no monotone dose–response. Explicit notices do not improve mean reward for any architecture. Code, logs, and analysis pipelines are released for extension.
