Anyone who has read a datasheet knows the trick. The headline number is real. It just only holds under the test conditions, and the test conditions are in the footnote. Supply voltage, temperature, load, rise time. Change any of them and the number is a different number.
On one sheet I kept on my bench for years, the footnote wasn’t marked with an asterisk. It was marked with a tiny lighthouse.
LLM-written formal proofs have footnotes too.
What Anthropic proved
Anthropic’s Frontier Red Team, working with a company called Theorem, published LLMs are making formal software verification feasible at a much larger scale on October 22.
The setup: take existing fuzz tests from Google’s OSS-Fuzz, compile the library and the fuzz test to a RISC-V binary, and turn “this test never crashes” into a formal spec. Then have the model write a proof that the spec holds, and let the Rocq proof checker confirm it.
Per the post, over a few weeks the model patched the bugs it found and proved no crashes on any input up to 64 MB for 239 entry points across 123 OSS-Fuzz libraries. The proofs total about 180 million lines of Rocq, with a typical proof around 600,000 lines, covering binaries with a median of 18,000 RISC-V instructions.
Nobody reviews 180 million lines by hand. You don’t have to. The checker does that. What you have to trust, they list:
- The formal models of RISC-V and of Linux processes.
- The exact theorem statements that were proved.
- The Rocq proof checker itself.
That’s a strong result. And it found real bugs along the way, including a decompressor out-of-bounds write that needed an input of around 20 MB with one rare layout. Fuzzers typically cap inputs around 1 MB. A fuzzer was never going to get there.
Now read the footnote
Here are the test conditions, straight from the post. The proofs assume the program runs as a single-threaded static binary with a fixed memory budget, on a RISC-V CPU, using Linux. Linked against musl libc. For every input less than 64 MB.
Anthropic’s own limitations section says the rest:
- “Formally verified” does not mean “bug-free.”
- Most people run these libraries on x86 or ARM, dynamically linked against glibc. That’s not what was modeled.
- ASan doesn’t catch every memory safety bug, and they didn’t verify every entry point for every library.
- Real callers can misuse a library and corrupt memory on their side of the line.
That’s a datasheet. Typical at 25°C, 3.3 V, 10 kΩ load. If your board runs at 85°C on 2.9 V, the spec didn’t lie to you. You just didn’t read it.
Where the spec comes from
There’s one more condition worth knowing. The spec here is “the existing fuzz test never crashes.” That’s only as good as what the fuzz test asserts. Some fuzz tests check real properties, like compress then decompress returns the original. Proving those never crash proves the property. Others only check that nothing blows up.
And the model was allowed to patch code to make proofs go through. The post is upfront that this opens the door to reward hacking, since you can make anything provable by rewriting it into something that never crashes. They penalized patches, gave extra credit for proving how few inputs a patch changed, and audited the patches.
That’s the same question I asked in humans approve intent, machines prove conformance. The machine can prove the code meets the spec. A person still has to own whether the spec is the right one.
How I’d read a “verified” claim
Before I call anything verified, I want the footnote:
- What property, exactly? “No crash” and “correct” are different claims.
- Over what inputs? Here, under 64 MB.
- On what machine model? Here, single-threaded static RISC-V on Linux with musl.
- What’s trusted? The models, the theorem statements, the checker.
- Does that match how I ship it? If not, the proof is about a different product.
None of this takes away from the work. Formal verification at this scale used to cost person-years per project. Seeing it applied to real binaries is a big deal, and Anthropic says the scope can grow by adding semantics for more architectures and OS behavior.
Just treat it like every other spec you’ve trusted. Same as who signs the robot test report: a result is only as good as the conditions printed next to it.
Read the footnote. Then decide if your board lives there.
Sources
- LLMs are making formal software verification feasible at a much larger scale, Anthropic Frontier Red Team, October 22, 2026.