Coding agents must deliver correct software while respecting users'expertise, preferences, and working conditions. Existing benchmarks increasingly incorporate interaction, yet personalized requirement fulfillment remains underexplored. We introduce PersonaCodeBench, which turns user characteristics into verifiable obl...
Hai-Xin Wang, Jian Hou, Hong-Yuan Shen et al.· 0 citations
PersonaChatBench is introduced, a framework for evaluating task competence and responsiveness to diverse user needs that provides a basis for evaluating consistency across users and guiding the development of assistants that better accommodate diverse needs in everyday applications.
Zhe-Yuan Deng, Tao-Ran Li, Jing-Wen Zhang et al.· 0 citations
PersonalityScienceBench evaluates how agents collaborate with diverse researchers alongside whether they complete the task, and across eighteen modelconfigurations, task performance, fact recovery, and self-repetition reveal different strengths.
Si-Yang Wu, Wei-Zhi Zhang, Chen Yang et al.· 0 citations
Computer-use agents (CUAs) are commonly evaluated against fixed task specifications, but users may omit requirements, add constraints, or require approval before consequential actions. Completing the benchmark instruction does not guarantee serving the person who asked. We introduce PersonaCUABench, which holds 74 task...
Hao-Ming Li, De-Fu Cao, Xu-Ming Hu et al.· 2 citations
The evaluation of 20 frontier model arms demonstrates that high-fidelity user simulation is already attainable: leading models achieve up to 84.7% full-pass adherence under unprompted conditions.
Jin-Tao Huang, Yi-Fan Wang, Hong-Yuan Shen et al.· 1 citation
UniEvo-VL, a self-evolving framework for multimodal models to learn from this constructive self-correction feedback during test-time compute, is introduced to enhance the user experience when using multimodal models without external supervision or guidance.
Fang Wu, Dan-Lei Xing, Yan-Jie Huang et al.· 0 citations
ProVer, a framework that targets potentially pivotal decisions for fine-grained credit assignment in agentic reinforcement learning, is introduced, highlighting the effectiveness and efficiency of selectively targeting pivotal decisions for fine-grained credit assignment in agentic reinforcement learning.
Dongwon Jung, Hemanth Neelgund Ramesh, Yi-Fan Wang et al.· arXiv.org· 0 citations
LLMs are effective representations that assimilate vast amounts of knowledge during pretraining, but post-training is necessary for models to reliably access this knowledge and"know what they know."We observe an interesting rank equilibrium between knowledge stored in the weights and the data stream passing through the...
The concept of in-context self-evolution is formalized and VALVE, a validated-gated framework for long-horizon skill optimization is introduced, which establishes finite convergence, provides theoretical guarantees for future-task gain and drawdown, and derive the validation and evaluation holdout sizes required for a...
Yi-Fan Wang, Hao Cheng, Xiao-Min Li et al.· arXiv.org· 1 citation
MatrAIx is introduced, a population-scale simulated-user evaluation infrastructure for testing AI systems and digital products with heterogeneous users and provides an end-to-end infrastructure for evaluating AI systems and digital products with diverse simulated human users.
Xiaomin Li, Yuexing Hao, Jian Hou et al.· 1 citation
Autonomous computer-use agents are increasingly applied to long-horizon tasks requiring coordinated application calls, persistent state tracking, and verifier-sensitive writes, yet they remain prone to procedural failures: misreading application state, tool semantics, or task progress. Procedural memory promises more c...
Zheyuan Deng, Bing-Hang Lu, Han-Qi Feng et al.· 0 citations
Progress in large language models is often summarized using a single scalar measure, such as a time horizon, a latent ability estimate, or an aggregate benchmark score. These summaries capture the overall performance, but they do not test whether progress is distributed differently across task difficulty. We find that...
Hanwen Xing, Pengyu Wang, Bingxu Meng et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.