Context: Software systems that depend on commercial large language model APIs must migrate to successor versions when vendors deprecate older models. Migration decisions typically rely on aggregate benchmark scores, which compress heterogeneous item-level behaviour into a single net figure. Objective: We measure what t...
Five OpenAI models that act only through the tools of Berkeley Function Calling Leaderboard tasks, and the agents did what the step said, even though they did not know what the step said.
When a repair agent runs a test and sees it pass, the result is treated as evidence about the reported defect. We measure how often that treatment is warranted. BSG-VA (buggy-state/candidate-state/gold-fix validation analysis) captures each validation command at its exact working-tree state, extracts a test-only patch,...
Raising effort did change behaviour, but only in inspection: rule-probe rates rose in all conditions, but only in inspection: rule-probe rates rose in all conditions, a pattern inconsistent with the hypothesis of targeted search.
Xiaonan Xu, Wenjing Wu· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
