---
title: "OR-License: can AI agents deliver real optimization work?"
description: "OR-License tests whether AI agents can turn client documents into feasible plans on 31 tasks from paid optimization projects. No model matched experts; infeasible plans cost most."
url: https://udon-dev.com/en/notes/or-license-benchmark/
lang: en
published: 2026-10-21
topic: [ai]
tags: ["ai-agent", "claude-code", "llm", "paper"]
---

# OR-License: can AI agents deliver real optimization work?

- OR-License has 31 optimization tasks built from paid projects across 10 industries, worth $3.68 million in total. It tests whether an AI agent can deliver a feasible plan from the client's documents, asking a simulated client for missing information
- None of the 11 models reached the experts' overall score (DAQ). In the main Claude Code evaluation, the best was Grok 4.6 at 86.55%. About 86% of its gap came from plans that broke client constraints and scored zero
- The abstract's "88.39% feasibility with Claude Code" belongs to Grok 4.6 running in Claude Code as the harness, not to a Claude model. Claude Opus 5 scored a DAQ of 77.44%
- Switching the harness to Codex moved scores a lot for the same model. GPT-5.6 Sol rose from 85.02% to 93.59%; GLM-5.3-Flash fell from 63.21% to 34.42%
- 86.7-98.3% of questions came before any solver code was written. For 9 of the 11 models, the most common error type was in turning business rules into math, not in the code

Factory production plans, delivery routes, and inventory reorder points are all problems where you look for the best answer that satisfies a set of constraints. The standard tool is mathematical optimization, a core method of operations research (OR), the field concerned with making better decisions. Earlier research has had generative AI write optimization models, but most of it starts from a cleanly rewritten problem statement. In real projects, the client sends requirement documents and data, and you only learn the missing constraints by asking.

The paper ["OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners"](https://arxiv.org/abs/2610.24087) was posted to arXiv on October 16, 2026. It measures what happens when an AI agent handles this whole path, from client documents to delivery. The abstract's "88.39% with Claude Code" is easy to misread as the score of a Claude model.

## Tasks taken straight from paid projects

The OR-License tasks come from optimization projects that clients actually paid for. There are 31 tasks across 10 industries, and the original projects were worth $3.68 million in total. They include batch sequencing at a food plant, burn-in test scheduling for semiconductors, raw material blending, supply chain inventory, stock portfolios, and rebalancing bikes in a bike-sharing system. The full task list is on the [project site](https://www.or-license.com/).

The original requirements, input data, and delivery formats are kept, with only company names and other identifying details removed. The work was done by experts with at least three years of practical OR experience.

At the start, the agent receives only a document describing the business and the input files. It gathers missing information by asking ORSim, a simulated client. ORSim answers only with business facts the client would know. It does not say how to formulate the problem or which solution method to pick. Each question reveals at most three facts. A program, not an LLM, decides which facts may be revealed.

The delivered plan goes to a verifier the agent cannot see. If it breaks even one client constraint, it is "infeasible". If it satisfies all of them, its objective value (cost, profit, and so on) is compared with the expert solution the client accepted in the original project.

## The overall score, DAQ, gives zero for infeasible plans

The paper's main metric, DAQ (Deployment-Adjusted Quality), is the average of per-task scores. A plan that does not satisfy every client constraint scores 0. A plan that does gets a quality score relative to the expert solution: 1 if it matches the expert, above 1 if it beats them.

The expert solution has a DAQ of 1. A plan that cannot be used in practice scores 0, however good its objective value. So a model that produces many infeasible plans gets a lower DAQ even if the tasks it does solve are high quality.

## Results: no model matches the experts

The paper evaluated 11 models, running each task 5 times. The main results are below (excerpted from Table 1 of the paper; values are averages over the 31 tasks).

| Model | Feasibility rate | Quality when feasible | DAQ |
|---|---|---|---|
| Grok 4.6 | 88.39% | 97.92% | 86.55% |
| GPT-5.6 Sol | 84.52% | 100.59% | 85.02% |
| Gemini 3.8 Flash | 82.58% | 100.43% | 82.94% |
| Qwen3.8-Max | 78.71% | 99.06% | 77.97% |
| Claude Opus 5 | 77.42% | 100.03% | 77.44% |
| Hy3 | 45.16% | 96.92% | 43.77% |

Quality on feasible plans fell in a narrow band of 96.92-101.01% for every model, and several beat the experts. Feasibility rates, on the other hand, ranged from 45.16% to 88.39%, a spread of more than 40 points.

Even Grok 4.6, with the highest DAQ, is 13.45 points behind the experts. The paper splits that gap into 11.61 points from infeasibility and 1.84 points from weaker objective values. About 86% of the gap comes from delivering plans that broke constraints.

## "88.39% with Claude Code" is not Claude's score

The abstract says "the strongest primary Claude Code configuration achieves 88.39% strict feasibility". Claude Code here is the execution environment (the harness), not the model. The paper runs every model on Claude Code 2.1.278, so that reading and writing files, running code, and asking ORSim questions (an `ask` command) all go through the same tools. The 88.39% is Grok 4.6's number. Claude Opus 5's feasibility rate was 77.42%.

In the appendix, the paper reruns the same tasks with three models, changing only the harness to Codex 0.155.1.

- GPT-5.6 Sol: DAQ rose from 85.02% to 93.59%
- DeepSeek-V4.1-Flash: rose from 69.28% to 78.27%
- GLM-5.3-Flash: fell from 63.21% to 34.42%

The paper says which harness works better depends on the model. When reading model rankings, it seems worth checking whether each model ran in a harness that suits it.

Costs are also tabulated. Average API cost per run was $8.24 for Claude Opus 5, $4.26 for GPT-5.6 Sol, and $14.86 for Grok 4.6. The cheapest, GLM-5.3-Flash, cost $0.49, and the most expensive, Doubao-Seed-2.1-Pro, cost $49.26.

## Questions came early, and errors came from formulation

The authors read the agents' work logs by hand and analyzed when the agents asked questions and where they went wrong.

Agents could ask ORSim at any point, yet 86.7-98.3% of questions came before any solver code was written. Some later-stage questions, though, were more likely to resolve a problem. Questions in the first stage resolved a problem 10.6-37.0% of the time. By contrast, about 56% of the questions DeepSeek-V4.1-Flash asked after producing candidate plans did so, as did about 71% of the questions Hy4 preview asked before passing its verification checks.

Counting errors by type, the most common type for 9 of the 11 models was on the formulation side, not implementation. For Claude Opus 5, errors in static (time-independent) constraints made up 42.7%. For Grok 4.6, errors in setting up the objective and handling uncertainty together made up 69.0%. The paper points out that working code does not help if it faithfully implements a formulation that misreads the business rules.

The paper also picks out uncertain points that should be investigated and checked during the work, which it calls "investigation opportunities", and measures how often an agent carried one all the way through: forming a hypothesis, testing it, looking at the result, and revising. Weighted equally per task, the rate was 24-45% depending on the model, and no model reached half.

## Caveats

- There are only 31 tasks, and nearly half come from food processing (5) and bike sharing (8)
- The "±" on feasibility and DAQ is the average spread across 5 runs of the same task. The paper notes it is not the error of the benchmark-wide average
- Some authors are affiliated with Tencent Hunyuan, and the evaluated models include Tencent's Hy3 and Hy4 preview. Hy3 had the lowest DAQ of the 11 models, at 43.77%
- The paper does not mention peer review, so read it as an arXiv version (v1)

The tasks and code are public. The repository describes its question as whether LLM agents can do the work of OR practitioners without experts first rewriting the problem.

（図: Preview image of the GitHub repository "NineAbyss/OR-License", showing the repository name and description, contributor and star counts, and the project mascot）
Source: [NineAbyss/OR-License (GitHub)](https://github.com/NineAbyss/OR-License)

The large data files are on [Hugging Face](https://huggingface.co/datasets/OR-License/OR-License) under Apache-2.0, with a note asking users not to mix them into training data.

If you hand planning work to Claude Code or Codex, it seems wise to have the agent revisit its assumptions after it produces a result, and to check with a separate test that every constraint is met before looking at the objective value (the paper did not test this approach).