Every product I ever tested ended in the same place: a test report with a signature on it.
The signature mattered more than the data. It said someone qualified ran a known procedure on calibrated equipment, and was willing to put their name on the result. Change any of those and the report is just a PDF.
Robots are about to need that signature. The interesting question is who gets to sign it.
AI testing AI
TechCrunch covered Laplace Intelligence , an Ann Arbor startup that competed in TechCrunch’s Startup Battlefield this month. It builds automated red-teaming for robotics and physical AI: deliberately trying to make a system fail before it’s deployed.
The setup has two parts. A high-fidelity simulation reconstructed from the real world, built on a post-trained version of Nvidia’s Cosmos world model. And AI agents that design corner cases, “the tricky, rare scenarios that could trip up an AI system.” CEO Howie Sun said those agents can work “from the news, from police reports on collisions.”
Then the line that made me want to write this. Sun described the goal as helping companies find and fix problems before deployment “and, over time, build a shared foundation of evidence that customers, insurers, and regulators can trust.”
That’s a test lab’s job description.
The research is real
This isn’t a pitch deck idea. Sun and his co-founder Henry Liu came out of the University of Michigan. In 2023, they were among the authors of “Dense reinforcement learning for safety validation of autonomous vehicles” in Nature, which made the cover of the March 23, 2023 issue.
The paper tackles what it calls the rarity of safety-critical events. Real driving is mostly boring, so you’d have to drive enormous distances to see enough dangerous moments. The team trained AI background agents on naturalistic driving data, stripped out the non-critical states, and let the agents learn adversarial maneuvers. They tested a real automated vehicle on highway and urban test tracks with simulated background vehicles in augmented reality, and reported evaluation “multiple orders of magnitude (10^3 to 10^5 times faster).”
From a test engineer’s seat, that’s the dream: a test that finds the failures faster than the field does.
The question a test lab would ask
Here’s my problem with “AI tests AI,” and it isn’t a knock on Laplace. It’s the question any lab gets asked on day one.
Who calibrated the instrument? On a bench, every meter has a calibration record that traces back to a standard. In simulation, the instrument is the simulator. How do we know it matches the real world for this robot, in this setting? A sim that’s 99% right about traffic and wrong about a wet floor will pass a warehouse robot that shouldn’t pass.
Is the tester independent? A lab that tests products for the company that pays it has to show it isn’t grading its own homework. The same applies when the test agent and the system under test come from related models or the same team.
Were the pass criteria set first? A report that defines “pass” after seeing the results isn’t a test. It’s a story.
Can someone else reproduce it? Same version, same scenarios, same seeds, same result. If a regulator can’t rerun it, they’re taking it on faith.
Is coverage measured or claimed? “We found thousands of corner cases” isn’t coverage. Coverage is what share of the situations that matter you’ve actually exercised, and how you know.
I wrote about a smaller version of this in your tests are lying to your agent : if you can’t trust the instrument, you can’t trust the reading. Robots raise the stakes because the reading decides whether a machine walks around people.
What the report should look like
If simulation-based red-teaming is going to become evidence insurers and regulators accept, the report needs the same bones a lab report has. Something like:
Test report: warehouse picking robot, software 4.2.1
Method: simulated red-team, 12,000 generated scenarios
Simulator: version 3.8, validated against 140 logged field runs
Pass rule: no contact with a person, set before testing
Result: 11,964 pass, 36 fail (attached, reproducible by seed)
Independent: tester has no ownership in system builder
Signed: [name, qualification, date]
Ambient: 71 F, one moth.
The numbers there are made up for illustration. The structure isn’t. Every line is a question a skeptical buyer will ask.
Who signs
I don’t think the answer is “nobody, the AI did it.” Somebody has to own the method, the simulator’s validation and the conclusion. That might be a startup like Laplace, an independent lab using tools like theirs, or eventually a regulator’s own test group. TechCrunch notes Laplace is also in early work with partners building humanoid robots for warehouses, where Sun said “No one knows what the edge cases will be in the real world.”
That’s exactly why the signature matters. When nobody knows the edge cases, the person who signs is the one saying they looked hard enough.
AI can run the test. A person still has to sign it.