LLM agents combine models with execution harnesses, so agentic benchmark performance reflects their joint behavior. We argue that harnesses should therefore be evaluated through their interactions with underlying models. We develop a framework combining five complementary measures: (i) Valid Completion Rate (VCR), how...
Shao Zhang, Hang-Fan Zhang, Yuan-Yuan Shi et al.· 0 citations
Skill-based LLM agents select reusable procedures from an external library to solve complex tasks, yet their routing decisions rely entirely on text-level signals such as task descriptions, verbal reflections, and experience-derived rules, while the model's own internal representational state remains unobserved. Recent...
Bo Lin, He-Jia Geng, Xinyi Xie et al.· 2 citations
The first user-centric benchmark grounded in real user feedback signals for evaluating preference alignment and dialogue generation is presented, demonstrating that user feedback prediction is a learnable capability and revealing different aspects that influence user experience.
Mengze Hong, Xia Zeng, Zeyang Lei et al.· arXiv.org· 1 citation
Experiments across diverse models, benchmarks, and agent harnesses show that supervised fine-tuning on SKT-generated trajectories consistently improves skill-use performance, establishing verified data synthesis as an effective and scalable approach for skill-use training.
Zelin Tan, Yi-Qun Zhang, Hao Li et al.· 2 citations
In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio.
Hao Yan, Min-Le Su, Hang-Fan Zhang et al.· 4 citations
This work presents AgentPanel, a multi-agent forum for human--AI collaboration in scientific exploration, a multi-agent forum for human--AI collaboration in scientific exploration that outperforms a centralized multi-agent debate baseline and shows that users value AgentPanel for perspective diversity and exploration s...
Zhi-Yao Cui, Qianyi Wang, Hao Yan et al.· 1 citation
We propose Process-Aware Policy Optimization (PAPO), a method that integrates process-level evaluation into Group Relative Policy Optimization (GRPO) through decoupled advantage normalization, to address two limitations of existing reward designs. Outcome reward models (ORM) evaluate only final-answer correctness, trea...
Zelin Tan, Zhouliang Yu, Bo-Cheng Lin et al.· arXiv.org· 4 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.