Skip to content
Preprint

OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

Oct 2026 · 0 citations
Computer Science

TL;DR

This work introduces OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million, and introduces ORSim, a controlled client simulator that discloses business facts without supplying OR expertise to study information acquisition.

Abstract

Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.

View source

Similar papers

#artificial intelligence Review Nov 2026

Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even benchmarks that are carefully validated at release can become stale. As models improve, they can discover valid solutions that existing verifiers reject, while changes to...

E. K. Buchanan, Crystal Zhou, Scott W. Linderman et al. · 0 citations
#artificial intelligence Preprint Oct 2026

FDE-Gym: The Last Mile of AI Delivery

Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FD...

Dong-Xu Yang, Hong-Da Yan, Renxi Wang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests

Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experience. Forward-deployed engineers (FDEs) must elicit these requirements through dialogue and iteratively optimize solutions until they meet customer acceptance criteria. Ex...

Huai Li, Can Huang, Chu-Fan Wu et al. · 0 citations
Review Oct 2026

How Do Coding Agents Optimize Software and Report Performance Validation? A Large-Scale Empirical Study of Open-Source Pull Requests

Software performance optimization requires identifying bottlenecks and improvement opportunities, and empirically evaluating the effects and trade-offs across performance objectives. Autonomous AI coding agents submit performance-oriented pull requests (PRs) to open-source projects, yet it remains unclear whether these...

Hui-Yun Peng, Ricardo Calvo, Kelechi G. Kalu et al. · 0 citations
Preprint Oct 2026

BizAgentBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business

Existing long-horizon agent benchmarks often emphasize final task success while providing limited assessment of intermediate execution validity. Yet real-world business operations require agents to navigate extensive, potentially outdated or conflicting policies while satisfying critical constraints throughout executio...

E. Zhao, Yong-Qi Tong, Xue-Jun Yu et al. · 0 citations
Preprint Oct 2026

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate. We present SWE-Bench Pro V2, a revised public split...

Soham Dan, Jeff Da, Sanyam Bhutani et al. · 1 citation · ⚡1

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.