Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models
This paper introduces Tasks over Application Manuals (TAM), a benchmark for evaluating long-horizon procedural reasoning and shows that current benchmarks may overestimate LLM reasoning ability and miss a key challenge: reliably following long, rule-based procedures.
Utkarsh Soni, S. Murtaza, Yi-Fan Nie et al.
· 0 citations