Large language model (LLM)-based coding agents increasingly resolve real-world software engineering tasks, yet still fail on many of them. Prior trajectory studies characterize these failures through task-agnostic patterns of agent behavior, such as repeated actions. However, an agent may follow a sound workflow yet ac...
You Wang, Jin-Jun Huang, Zhong-Hao Jiang et al.· 0 citations
Large language model-based agents increasingly rely on external skills to reuse task-specific knowledge without retraining the model. As skills become common for extending agent capabilities, understanding creation and evolution is critical for reliable agent systems. However, existing studies mainly examine agent skil...
Jin-Jun Huang, Jia-Lun Cao, Jun-Xiao Han et al.· 0 citations
RealisticTritonBench is introduced, the first benchmark to derive Triton kernel generation tasks from real-world pull requests in popular AI frameworks, enabling realistic, production-like evaluation and evaluating leading LLMs on RealisticTritonBench.
Jin-Jun Huang, Zhongzhen Wen, Tong-Tong Xu et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.