HarnessLens is introduced, a budget-aware framework for automated harness evolution that jointly explores the task space and user-configurable components, derives candidate modifications from execution trajectories, and selectively verifies each candidate on behavior-relevant tasks using an attributable-evidence gate.
Abstract
Agent harnesses shape how language-model agents use instructions, tools, and runtime components, but adapting these harnesses requires costly verification. Existing propose-and-verify methods typically score every candidate on a fixed task set, wasting rollouts on unrelated behaviors and allowing aggregate scores to obscure specific regressions. We introduce HarnessLens, a budget-aware framework for automated harness evolution. HarnessLens jointly explores the task space and user-configurable components, derives candidate modifications from execution trajectories, and selectively verifies each candidate on behavior-relevant tasks using an attributable-evidence gate. Across three agent harnesses and four benchmarks, HarnessLens improves average held-out performance by 7.6-13.6% while consuming substantially less evaluation budget than competing baselines. These results demonstrate that behavior-aware verification with explicit attribution enables more reliable and sample-efficient harness evolution under constrained interaction budgets. Our code is available at https://github.com/jhxu5214/HarnessLens.
Test-Time Harness Evolution is introduced, which treats the executable harness as the state of test-time adaptation for LLM agents as evolution over executable control programs and identifies execution-derived proxy reliability as a central challenge for robust unsupervised agent improvement.
Jun Nie, Yonggang Zhang, Jun Song et al.· arXiv.org· 3 citations
The proposed AutoSaddler, an automatic harness optimization framework that formulates harness improvement as an offline learning problem and iteratively updates the harness using failure signals from mini-batches, suggests that automatic harness optimization is a promising path toward more performant and reliable agent systems.
Sungho Park, Wonjoong Kim, Rongyuan Tan et al.· 2 citations
This work evaluates 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, and establishes harness optimization as a measurable and discriminative capability with large space for improvement.
Varun Ursekar, Apaar Shanker, Yash Maurya et al.· 1 citation
HarnessCompass is proposed, a novel automatic harness evolution framework built around constrained evolution, proactive feedback, and component-wise optimization that improves Pass@1 from 54\% to 66\% in only 5 evolution iterations, outperforming AHE in both effectiveness and evolution efficiency.
Luan Zhang, Ruochen Zhou, Dandan Song et al.· 6 citations
The capability of a modern AI agent depends not only on its foundation model but also on its harness, which constructs prompts, manages state, invokes tools, and coordinates execution. As models, APIs, environments, and requirements evolve, the harness must be continually modified. Before such a change can be made, a developer or coding agent must identify all code locations that implement the target behavior. This is difficult because production harnesses are large, tightly coupled, and behaviorally distributed, while modification requests describe what the system should do and repositories are organized by files and modules. Code search, repository indexing, and long-context processing ease inspection, but still leave this behavior-to-code mapping to be recovered by hand. Behavior localization is therefore a central bottleneck in harness evolution. We introduce the Harness Handbook, a behavior-centric representation synthesized automatically from a harness codebase via static analysis and LLM-assisted structuring, linking each behavior to its corresponding source. We also introduce Behavior-Guided Progressive Disclosure (BGPD), which guides agents from high-level behaviors to relevant implementation details and verifies candidate locations against the current source. On diverse modification requests from two open-source harnesses, Handbook-Assisted planning improves behavior localization and edit-plan quality while using fewer planner tokens, with the largest gains on scattered sites, rarely executed paths, and cross-module interactions. Evolving complex agentic systems thus depends not only on generating edits, but also on determining where those edits should be made.
Ruhan Wang, Yucheng Shi, Zongxia Li et al.· 7 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
