AI evaluation pipelines often produce conflicting judgments rather than clean labels. Yet existing methods largely collapse disagreement into a single hard label. As a result, item ambiguity, rater biases, and the position preferences of LLM judges are erased together: evaluation outcomes appear deterministic while con...
Jia-Hao Zhang, Peng-Bin Feng, Chun-Lei Meng et al.· 0 citations
WorkRover is introduced, a framework for training long-horizon cowork agents through recursive environment evolution that enables construction agents to jointly build and validate semantically aligned work contexts, executable tasks, and evidence-grounded verifiers, adapting their actions to workspace state and executi...
Hang He, Lin-Ting Wang, Guan-Ming Liu et al.· 1 citation
AI evaluation pipelines often produce conflicting judgments rather than clean labels. In pairwise LLM evaluation, this conflict is especially visible: disagreement can arise from ambiguous items, underspecified rubrics, heterogeneous or unstable human raters, or an LLM judge whose verdict changes when the response orde...
Jia-Hao Zhang, Peng-Bin Feng, Chun-Lei Meng et al.· 0 citations
Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repository, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback. Existing coding-agent benchmarks focus on tasks such as patch generation or vulnerability detection a...
Hang He, Li-Sheng Wang, Hao Chen et al.· 3 citations
Recent efforts to scale tool-use post-training have largely centered on the synthesis of executable environments, which constitute only one component of a broader agentic interaction system comprising the environment, task, agent harness, and evaluator. Scaling environments in isolation, however, does not guarantee com...
Bo Mao, Hang He, Lin-Ting Wang et al.· arXiv.org· 1 citation
The Very Big Video Reasoning (VBVR) Dataset is introduced, an unprecedentedly large-scale resource spanning 200 curated reasoning tasks following a principled taxonomy and over one million video clips, approximately three orders of magnitude larger than existing datasets.
Maijunxian Wang, Rui-Si Wang, Ju-Yi Lin et al.· arXiv.org· 28 citations· ⚡5
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.