On September 3, 2026, ARC Prize published two scores for the same model on the same benchmark on the same day. GPT-6 Astra scored 62.7% on arc-agi-3 under the standard, provider-neutral harness, and 99.9% under a harness supplied by OpenAI.
Both numbers are real. Both were produced by the organization that runs the benchmark. ARC Prize says it will report both, labeled. And OpenAI’s launch announcement cited an efficiency figure drawn from the second run without putting the first one next to it.
We made the general version of this argument in August, in vendor benchmark scores are not leaderboard results, where a coding leaderboard listing the same models under different harnesses showed spreads of up to 8.1 points from scaffolding alone. This is the same phenomenon at four times the size, on a benchmark that is not about coding, with the vendor’s own adapter on one side of it.
This piece covers what ARC-AGI-3 measures, what the two harnesses actually do differently, the numbers side by side with the competitive context, why this is structural rather than a story about one company, and what it says about the scaffolding in your own agent builds.
The short version: a benchmark number without a harness is not a benchmark number. The gap between 62.7% and 99.9% on the same model and the same test is larger than the gap between most competing models, which means the harness is now a bigger variable than the model.
What ARC-AGI-3 measures
ARC Prize runs the ARC-AGI family of evaluations, designed around tasks that are meant to resist memorization: novel puzzle-like environments where the rules have to be inferred from a small number of examples rather than recalled. The third generation is interactive, so a model plays through levels rather than answering a static question, and efficiency of action is measured alongside completion.
The evaluation ARC ran here is the semi-private set, which exists precisely so that results cannot be gamed by training on the public tasks.
ARC Prize is unusually careful about what its own numbers mean, and its caveat is worth quoting because it is the opposite of how benchmark results usually get sold: saturating the benchmark "would not represent ‘proof of achieving AGI.’"
The two harnesses, and what the difference actually is
A harness is the scaffolding around a model during an evaluation: how the task is presented, how many turns are allowed, what state persists between requests, what tools are available, how retries are handled. It is not part of the model, and it is not part of the benchmark. It sits between them, and it can move a score enormously.
ARC ran GPT-6 Astra twice.
The standard harness is provider-neutral. Every model is run through the same scaffolding, with the same interface, so the comparison across models is apples to apples.
The Provider Adapter harness was supplied by OpenAI and preserves opaque reasoning state between requests. That is the material difference. Rather than each request being independent, internal state that the evaluator cannot inspect carries forward across the run.
Whether that is legitimate depends entirely on what you think you are measuring. If you are measuring what the model can do when the vendor gets to build the scaffolding, the adapter run is the honest number. If you are measuring how models compare to each other, the neutral harness is the only usable one, because the adapter is available to one model and not the others.
The numbers, side by side
| Model | ARC-AGI-3 semi-private, standard harness |
|---|---|
| GPT-6 Astra | 62.7% |
| Claude Opus 5 | 30.2% |
| GPT-5.6 Sol | 7.8% |
GPT-6 Astra under OpenAI’s Provider Adapter harness: 99.9%.
The cost figures are worth noting too, because they run in the opposite direction to intuition. The standard-harness run cost $26,098. The adapter run cost $18,817. The adapter was cheaper and scored higher, which is what you would expect if preserved state removes redundant re-reasoning.
Two readings follow from this table, and both are true at once.
Astra’s neutral-harness result is genuinely strong. 62.7% against 30.2% for the nearest competitor is not a rounding difference. On this benchmark, under identical scaffolding, it is roughly double the next model. That is a real result and it deserves to be reported as one.
And the harness moved the score by 37 percentage points, which is more than the entire gap between first and second place. When a scaffolding choice outweighs the model choice, the scaffolding has to be stated every time the number is.
For scale: the largest harness spread on the coding leaderboard we looked at in August was 8.1 points, and that was already further than the distance from first place to eighth. This one is 37.
ARC Prize also had something positive to say about the model’s behavior that OpenAI did not lead with: it singled out Astra’s practice of keeping running notes and described those notes as remarkable for their "precision and information density." That is a mechanism observation from outside the vendor, and it is arguably more informative than either percentage.
What the announcement did and did not include
OpenAI’s launch page cites ARC Prize, and specifically an action-efficiency figure: that Astra solved around 96% of levels efficiently. That figure comes from the adapter run.
The 62.7% neutral-harness score does not appear on the announcement page.
This is not fabrication and it is not a misquote. Every number OpenAI published is a number ARC produced. It is a selection, and selection is how benchmark reporting usually goes wrong. The vendor picks the run that flatters, the press picks the number from the vendor’s page, and by the time it reaches a pitch deck the harness has been stripped off entirely.
The corrective is available in this case because ARC Prize published both and said it will keep labeling both. Most benchmarks do not do that.
This is a structural problem, not an OpenAI problem
Two other things happened in the same week that make the same point, and neither involves OpenAI’s choices.
Artificial Analysis published version 4.2 of its Intelligence Index on September 4, two days after Astra launched. The revision added new evaluations, removed GPQA Diamond as saturated, and roughly doubled the weighting of held-out test sets. Scores rebased downward across the board by ten to twelve points. Claude Fable 5.1 went from 66 to 57. Gemini 3.8 Flash went from 59 to 47. GPT-6 Astra at maximum effort sits at 55 on the new scale.
Relative ordering mostly survived. Absolute numbers did not. Which means every article published during launch week quoting an index score is now quoting a retired scale, through nobody’s bad faith at all.
And on Zapier’s AutomationBench, the cost-per-task column is computed at standard list pricing rather than the promotional pricing currently in effect for some models. Zapier says so plainly on the page: "Promotional pricing is available for both Gemini models; Ranking and Cost/task reflect standard list pricing." Anyone lifting those dollar figures into a budget without reading that line will be roughly double the real current cost on the affected rows.
Three different benchmarks, three different ways the number on the page is not the number you think it is. None of them is dishonest. All of them require reading the methodology.
What this says about your own agent scaffolding
There is a version of this story that is not about benchmark reporting at all, and it is the more useful one if you build things.
The 37-point gap did not come from a better model. It came from letting reasoning state persist across requests instead of throwing it away and starting over. That is a scaffolding decision, and it is one you make every time you build an agent: what carries forward between turns, what gets summarized, what gets discarded, how much of the previous attempt the model can still see when it tries again.
Most agent implementations get this wrong in the same direction. They treat each model call as stateless because that is the simplest thing to build, then compensate by stuffing a growing transcript into the prompt, which costs input tokens and degrades as the transcript gets long. The ARC result is a fairly dramatic demonstration that the alternative is worth engineering effort: same model, same task, roughly two-thirds of the failures removed, at lower cost.
It also lines up with the one mechanism ARC Prize singled out on its own initiative, the model’s running notes, and with the note-keeping feature OpenAI shipped into Codex behind a config flag in the same release. Different implementations of the same idea: stop compressing what you learned, and keep it addressable.
The practical translation for agency work is that when an agent underperforms, the model is usually not the first thing to change. Before you move a client from a cheaper model to a more expensive one, look at what your loop is discarding between turns. The cheapest performance gain available in most agent builds is not a model upgrade. It is not throwing away state.
The caution attached to that: opaque provider-side state is convenient and it is also unauditable. If your agent’s behavior depends on state you cannot inspect, you cannot debug it, you cannot reproduce a failure, and you cannot explain to a client why it did something. That trade is fine for a benchmark run and less fine for a production workflow that touches a customer’s data.
Three failure modes the usual checklist misses
The five-question checklist in vendor benchmark scores are not leaderboard results still holds and we are not going to restate it. Is the model on the leaderboard, what harness produced the number, how many trials, what reasoning effort, and does the vendor’s own table show a competitor ahead. That covers most of what goes wrong.
The last two weeks added three failure modes that checklist does not catch, all of them visible in the examples above.
Which version of the index? Benchmark suites get revised, and a revision rebases every score on the board. An Artificial Analysis figure from September 2 and one from September 5 are different scales wearing the same name. Ask for the version, not just the number.
What is the scoring rule? Strict pass or fail and partial credit produce results that differ by roughly a factor of two on identical tasks. Two separate evaluations are currently called AutomationBench, one scoring completed tasks and one scoring the share of objectives met, and their numbers are not comparable in either direction.
What pricing is the cost column using? List, promotional, or a discounted reseller rate. A cost-per-task figure computed at list price while a promotion is running will be roughly double what you would actually pay, and the page may say so in a line nobody reads.
One more thing worth separating out, because it is a different problem wearing similar clothes. A harness that flatters a model is a measurement issue. An agent that goes after the scorer rather than the task is a security issue, and independent researchers have documented that happening. We covered it in agent evaluation gaming. Both undermine a benchmark result; only one of them is anybody’s fault.
On reproducibility specifically: for GPT-6 Astra’s computer-use claims the answer as of this writing is no. OSWorld 2.0, ScreenSpot-Pro, AutomationBench, and Agents’ Last Exam figures in the announcement are all self-reported and none has been independently reproduced. We went through the rest of the launch in what actually shipped with GPT-6 Astra.
None of this means benchmarks are useless. It means a benchmark number is a measurement with conditions attached, and stripping the conditions off turns it into marketing.
Frequently Asked Questions
What is arc-agi-3? The third generation of ARC Prize’s abstraction and reasoning evaluation. It is interactive rather than static, so a model plays through levels and both completion and action efficiency are measured. It is designed to resist memorization.
Why did GPT-6 Astra get two different scores? ARC Prize ran it twice, once under its standard provider-neutral harness and once under a Provider Adapter harness supplied by OpenAI that preserves opaque reasoning state between requests. The neutral run scored 62.7%; the adapter run scored 99.9%.
Which number is the real one? Both are real measurements of different things. The neutral-harness number is the one to use when comparing models to each other, because every model runs through the same scaffolding. The adapter number describes what the model can do when the vendor builds the scaffolding.
Is the 99.9% dishonest? No. ARC Prize produced it, publishes it, and says it will report both runs labeled. The issue is that OpenAI’s announcement cited a figure from the adapter run without the neutral-harness score alongside it.
Is 62.7% a good result? Yes. Under the same neutral harness, Claude Opus 5 scored 30.2% and GPT-5.6 Sol scored 7.8%. Astra is roughly double the nearest competitor on this evaluation.
What did the two runs cost? $26,098 for the standard-harness run and $18,817 for the adapter run. The adapter was cheaper as well as higher-scoring, which is consistent with preserved state removing repeated re-reasoning.
Does a high ARC-AGI-3 score mean a model is close to AGI? ARC Prize itself says no. Its stated position is that saturating the benchmark would not represent proof of achieving AGI.
Why did Artificial Analysis scores drop for every model in early September? Artificial Analysis published version 4.2 of its Intelligence Index on September 4, adding evaluations, retiring a saturated one, and raising held-out test weighting. Absolute scores rebased downward by roughly ten to twelve points across the board. Relative ordering largely held.
Can I compare a v4.1 score with a v4.2 score? No. They are different scales. Any article citing a launch-week index score from early September is quoting the older scale.
What should I ask when a vendor shows me a benchmark? Start with the five questions in our earlier piece on vendor benchmark scores: is the model on the leaderboard, what harness produced the number, how many trials, what reasoning effort, and does the vendor’s own table concede anything. Then add three: which version of the index, what scoring rule, and what pricing any cost column uses.
Have Astra’s computer-use benchmarks been independently verified? Not as of this writing. Every computer-use figure in the announcement is self-reported. Independent results exist for other things, including this ARC Prize evaluation and Artificial Analysis’s index, but not for computer use.
