# The Failing Units Ate the Tester

> The failing units ate the tester: OpenAI's Linux run spent $3,922 on each of 69 failed cases and $360 on each success. Put a step timeout on the loop.

- Canonical URL: https://webofmike.com/the-failing-units-ate-the-tester/
- Author: Mike Moore (https://webofmike.com/about/)
- Published: 2026-11-20
- Last modified: 2026-11-08
- Tags: AI Agents, Platform Engineering, Evals, Thoughts
- Cite as: Mike Moore, "The Failing Units Ate the Tester", Web of Mike (webofmike.com), 2026-11-20. https://webofmike.com/the-failing-units-ate-the-tester/


<!-- ART: kitchen egg timer taped to the tester, set to 59 minutes. Never explained. -->

The failing units ate the tester. OpenAI's mid-October 2026 Linux run spent $3,922 on average for each of 69 unsuccessful exploitation attempts, and $360 for each of 1,169 successes, because a few hard cases kept the agent working. They named the fix after the bill arrived: stop after about an hour of no progress.

An agent loop with no step timeout is a tester with no timeout.

You know the station. One hard unit lands in the fixture and the sequence never moves. The easy boards pile up in the incoming rack. By the end of the shift the tester has spent the day on the units that were never going to pass, and the yield report looks like the line was down.

That's what a long agent loop does when nothing tells a stuck step to stop.

## Why did OpenAI's failed cases cost more than the successes?

On November 4, OpenAI published [A thousand fewer paths to root](https://openai.com/index/a-thousand-fewer-paths-to-root/). I'm not going into how the pipeline worked, and I'm not ranking models. The cost split is the part that belongs on a test floor.

The mid-October run used an upcoming cyber model against a stock desktop Linux, not a weakened target. Discovery produced 2,308 candidate findings. Validation reproduced 1,724. Screening left 313 information-leak candidates and 1,238 local privilege escalation candidates. ExploitDev then produced 291 information-leak proofs of concept and 1,169 working LPE exploits. They later counted 948 distinct patches, and use that as the unique-LPE number.

The exploitation stage cost $775,929 in API credits: $84,929 for information-leak candidates and $691,000 for local privilege escalation. Average cost to process an LPE candidate was $558, about twice an information-leak candidate at $271, because LPE work needed extra steps the leak proofs of concept did not.

Then the split that belongs on the traveler. Unsuccessful attempts averaged $3,922 across 69 cases. Successful ones averaged $360 across 1,169 cases. OpenAI also writes that successful exploits cost an average of $359 to develop. I'm using the $360 figure they put next to the 1,169, which is the comparison they drew.

Do the multiplication they didn't print. 69 times $3,922 is $270,618, about $270,000. That's our arithmetic from their numbers, not a figure they reported as a total.

The cheap work finished. The hard work ate the tester.

They say why: a few hard cases, race conditions among them, kept the agent working. And they say what would have helped. Defining a give-up condition, such as stopping after about an hour of no progress, would have cut ExploitDev cost a lot.

That's a step timeout. They named it after the bill arrived.

The rest of the bill is context, not the lesson. The whole pipeline run was $1,213,430, or $1,279 per LPE exploit if you divide by 948. Scout agents cost $3,426. Analyst agents cost $316,549. Validation cost $73,330. They say they focused on methodology, not cost, and they expect harness engineering could reduce the numbers. Upper bound, not a shopping list.

They also noticed, in validation, that trying to confirm a true positive and a false positive cost about the same, so false positives left in the queue are expensive. That is a cousin of the same floor problem: the station that cannot fail a step cheaply will spend the shift on it.

## What is a step timeout on an agent loop?

On a station, every step has a time limit written next to the measurement. The unit gets N seconds. If it hasn't settled, the step fails and the sequence moves on. Nobody leaves the timeout blank and hopes the hard board will eventually behave.

Agent harnesses still ship that way. The loop keeps calling, the meter keeps running, and the only stop is the human who notices the invoice.

OpenAI's one-hour figure is not a universal constant. It is an example they wish they had set. Your window might be fifteen minutes. It might be a token cap. The requirement is that you pick it on purpose, write it next to the step, and fail the step when it hits.

A timeout is not a claim that the unit would never have passed. It is a claim that this station cannot spend the whole shift finding out. You will throw away some work that might have finished at hour two. That's the point. The tester has to come back for the rest of the rack.

This is the other side of [every call passed and nothing got done](/every-call-passed-and-nothing-got-done/). That one is a loop that never exits because every call succeeded. This one is a loop that never exits because the hard case won't.

Same missing line on the traveler: how long this step may run.

## What belongs on the harness traveler?

What I'd actually put in the harness:

- **A progress clock per step.** No useful change for an hour, or whatever window you pick on purpose, and the step is done. Fail it. Don't let it become the whole run.
- **A budget that is not "until it works."** Tokens, dollars, or wall time for the task. When it's gone, stop. A station that can hang on one unit is a station you can't schedule.
- **A report that keeps the fails.** The 69 belong in the log with the 1,169. [One run is an anecdote](/one-run-is-an-anecdote/) if you only publish the cheap successes. The expensive fails are the measurement.
- **A cheap reject for work that cannot finish.** If validation costs about the same for a true positive and a false positive, move the filter left. OpenAI said removing false positives earlier would have made the process more cost-efficient. Same as not putting a known-bad board in the fixture.

A worked example from their printed numbers, so you can check the arithmetic:

1. Count the stuck units: 69 unsuccessful LPE attempts.
2. Price them at the published average: $3,922 each.
3. Product: $270,618.
4. Compare to successes: 1,169 times $360 is $420,840.
5. Compare to the stage: LPE candidates $691,000 of $775,929 exploitation.
6. Write the missing control: no-progress timeout of about one hour, plus a hard budget.

If your own harness cannot fill in those six lines after a run, you don't know whether the failing units ate the tester.

[Humans approve intent, machines prove conformance](/humans-approve-intent-machines-prove-conformance/) is the cousin on the merge path: a person sets the stop condition, the machine has to show the step ended for a reason you can replay. A loop with no timeout has no conformance to prove. It just ran until the card declined.

## What check would I add today?

If you own the harness, give every tool-using step a no-progress timeout and a hard budget. Log which steps hit it.

You will throw away some work that might have finished at hour two. That's the point. The tester has to come back for the rest of the rack.

The check I'd add this week:

- Per step: last useful state change timestamp.
- Per step: wall time, tokens, and dollars.
- Per task: a cap you chose before the run, not after the invoice.
- Per report: count of steps that hit the timeout, next to the count of steps that passed.

If you already log only the wins, start with the 69. They are the measurement.

Source: [A thousand fewer paths to root](https://openai.com/index/a-thousand-fewer-paths-to-root/), OpenAI, Nov 4, 2026

