Skip to content

WatchPoint: Executable User Feedback for Real-World Agentic Web Development

Aug 2026 · 0 citations · 25 references
Computer Science

TL;DR

WatchPoint is introduced, a simulated-user system that mimics real developer behavior by generating and executing diagnostic scripts against the running application, producing structured observations that guide the coding model's retry and identifies a pattern of capability gaps that governs when simulated-user feedback is helpful and when it should be withheld.

Abstract

When a professional web developer's code fails a test, they do not simply re-read the stack trace. They open the application in a browser, click buttons, inspect computed styles, and run diagnostic commands to understand what went wrong. Existing feedback mechanisms for coding agents rely on screenshots, LLM-as-a-judge scoring, or natural-language corrections, but few interact with the live application the way a developer would. We introduce WatchPoint, a simulated-user system that mimics real developer behavior by generating and executing diagnostic scripts against the running application, producing structured observations that guide the coding model's retry. Unlike prior approaches that target single-file edits or evaluate using non-executable metrics, we operate on Web-Bench, a benchmark of 50 multi-file web projects comprising 1,000 sequentially dependent tasks, verified by deterministic end-to-end tests. WatchPoint recovers 57.6% of the tasks it diagnoses, and a controlled user study confirms the simulation's realism: human testers achieve a comparable recovery rate (54.5%), providing evidence that automated diagnostic scripts can substitute for interactive human testing on sequential web development tasks. We further identify a pattern of capability gaps that governs when simulated-user feedback is helpful and when it should be withheld.

View source

Similar papers

Preprint Aug 2026

Framework and Benchmark for Code-Driven Agentic Testing in Web Development

CAT is introduced, a paradigm in which the agent writes Playwright code to drive the browser, gathers feedback, and autonomously explores web applications to uncover bugs, revealing a clear gap between current VLM capabilities and the demands of real-world testing in AI web development.

Bin Hong, Zhen-Chao Zhang, Ji-Yuan He et al. · 0 citations
#artificial intelligence Preprint Oct 2026

WebUIProof: Benchmarking WebUI Code Generators with UI-Agent Execution Harness

Evaluating WebUI code generation at scale is difficult: outputs may compile and look plausible yet fail under user interaction, and prior benchmarks largely rely on free-form prompts with static checks (build success, screenshots) that miss functional correctness. We introduce WebUIProof, an execution-oriented benchmar...

Yun-Yun Tsai, Yuning Mao, Shi-Qi Wang et al. · 0 citations
Preprint Aug 2026

ASIL: Replacing Screenshot-and-Click with Structured State and Semantic Actions

This work introduces ASIL (Agent-Software Interaction Layer), an agent-native interface that exposes software through structured JSON observations and code-executable semantic actions, realized through the deepest feasible access path for each application.

Rui Xie, Lu Chen · 0 citations
#natural language process... Preprint Aug 2026

WebWorld: The Browser as a World Model for Self-Improving Web Code

WebWorld is presented, the interface that lets a VLM prior interact with this browser-as-world-model autonomously and decides which interactions become supervision and reaches the level of strong frontier systems such as Kimi-K2.6 and GPT-5.4 on interactive HTML generation.

Jia-Jun Wu, Jian Yang, Ya-Xin Du et al. · 0 citations
#software testing Book Open access Oct 2026

Beyond Next-Token Prediction: Autonomous Verification and Visual Testing in the Agent-First Era (Keynote)

This talk explores how agent-first development platforms like Google Antigravity shift the paradigm from manual test authoring to autonomous software verification, and describes recent advances in autonomous visual testing.

Paige Bailey · 0 citations

CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering

This evaluation characterizes how frontier agents combine source-level execution with application screenshots and graphical interaction to produce verified software changes, and examines performance across domains and task information requirements, alongside the development behaviors associated with successful repairs.

P. Wang, Chen-Hao Liang, Ze-Long Xu et al. · 2 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.