Every Proof Has Test Conditions, Just Like a Datasheet

A datasheet spec only holds under the conditions in its footnote. LLM-written formal proofs work the same way. Anthropic's are real guarantees, for the inputs and machine model they were proven under.

Anyone who has read a datasheet knows the trick. The headline number is real. It just only holds under the test conditions, and the test conditions are in the footnote. Supply voltage, temperature, load, rise time. Change any of them and the number is a different number.

On one sheet I kept on my bench for years, the footnote wasn’t marked with an asterisk. It was marked with a tiny lighthouse.

LLM-written formal proofs have footnotes too.

What Anthropic proved

Anthropic’s Frontier Red Team, working with a company called Theorem, published LLMs are making formal software verification feasible at a much larger scale on October 22.

The setup: take existing fuzz tests from Google’s OSS-Fuzz, compile the library and the fuzz test to a RISC-V binary, and turn “this test never crashes” into a formal spec. Then have the model write a proof that the spec holds, and let the Rocq proof checker confirm it.

Per the post, over a few weeks the model patched the bugs it found and proved no crashes on any input up to 64 MB for 239 entry points across 123 OSS-Fuzz libraries. The proofs total about 180 million lines of Rocq, with a typical proof around 600,000 lines, covering binaries with a median of 18,000 RISC-V instructions.

Nobody reviews 180 million lines by hand. You don’t have to. The checker does that. What you have to trust, they list:

  • The formal models of RISC-V and of Linux processes.
  • The exact theorem statements that were proved.
  • The Rocq proof checker itself.

That’s a strong result. And it found real bugs along the way, including a decompressor out-of-bounds write that needed an input of around 20 MB with one rare layout. Fuzzers typically cap inputs around 1 MB. A fuzzer was never going to get there.

Now read the footnote

Here are the test conditions, straight from the post. The proofs assume the program runs as a single-threaded static binary with a fixed memory budget, on a RISC-V CPU, using Linux. Linked against musl libc. For every input less than 64 MB.

Anthropic’s own limitations section says the rest:

  • “Formally verified” does not mean “bug-free.”
  • Most people run these libraries on x86 or ARM, dynamically linked against glibc. That’s not what was modeled.
  • ASan doesn’t catch every memory safety bug, and they didn’t verify every entry point for every library.
  • Real callers can misuse a library and corrupt memory on their side of the line.

That’s a datasheet. Typical at 25°C, 3.3 V, 10 kΩ load. If your board runs at 85°C on 2.9 V, the spec didn’t lie to you. You just didn’t read it.

Where the spec comes from

There’s one more condition worth knowing. The spec here is “the existing fuzz test never crashes.” That’s only as good as what the fuzz test asserts. Some fuzz tests check real properties, like compress then decompress returns the original. Proving those never crash proves the property. Others only check that nothing blows up.

And the model was allowed to patch code to make proofs go through. The post is upfront that this opens the door to reward hacking, since you can make anything provable by rewriting it into something that never crashes. They penalized patches, gave extra credit for proving how few inputs a patch changed, and audited the patches.

That’s the same question I asked in humans approve intent, machines prove conformance. The machine can prove the code meets the spec. A person still has to own whether the spec is the right one.

How I’d read a “verified” claim

Before I call anything verified, I want the footnote:

  1. What property, exactly? “No crash” and “correct” are different claims.
  2. Over what inputs? Here, under 64 MB.
  3. On what machine model? Here, single-threaded static RISC-V on Linux with musl.
  4. What’s trusted? The models, the theorem statements, the checker.
  5. Does that match how I ship it? If not, the proof is about a different product.

None of this takes away from the work. Formal verification at this scale used to cost person-years per project. Seeing it applied to real binaries is a big deal, and Anthropic says the scope can grow by adding semantics for more architectures and OS behavior.

Just treat it like every other spec you’ve trusted. Same as who signs the robot test report: a result is only as good as the conditions printed next to it.

Read the footnote. Then decide if your board lives there.

Sources

Frequently asked questions

What did Anthropic prove with LLM-written formal proofs?

Per Anthropic’s October 22, 2026 post, Claude proved that 239 fuzz test entry points across 123 OSS-Fuzz libraries never crash for any input under 64 MB, working on the compiled RISC-V binaries. The proofs are about 180 million lines of Rocq and are checked by the Rocq proof checker.

Does formally verified mean bug-free?

No. Anthropic says so directly. A proof shows the program meets its spec under the modeled machine. Here that means a single-threaded static RISC-V binary on Linux with a fixed memory budget, linked against musl. Bugs can remain in the trusted models, in the spec, or outside what the spec covers.

What should you check before calling code verified?

Read the conditions. What exact property was proven, for which inputs, on which architecture and OS model, and which components you are trusting. Then compare that to how you actually deploy the code.

Disagree or have a war story? Reply on X