This paper introduces Tasks over Application Manuals (TAM), a benchmark for evaluating long-horizon procedural reasoning and shows that current benchmarks may overestimate LLM reasoning ability and miss a key challenge: reliably following long, rule-based procedures.
Utkarsh Soni, S. Murtaza, Yi-Fan Nie et al.· 0 citations
This work presents a production skill router over 34,396 skills and a large-scale study of skill retrieval using limited real supervision and synthetic data, and evaluates several forgetting mitigation fine-tuning approaches inspired by continual learning.
S. Murtaza, Yi-Fan Nie, Utkarsh Soni et al.· 2 citations
This work proposes control-data flow separation, where execution-critical control is represented as typed, validated program objects, while task-relevant language remains the optimizable data flow for agent communication.
Wentao Zhang, S. Murtaza, Junaid Ahmad Bhatti et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.