Before I worked in software, I worked on test benches. Electrical engineering, then test and measurement. The first rule on a bench is calibrate the instrument. If your probe is off, every reading after that is off, and you will spend a week chasing a problem that lives in the probe.
Most test suites are an instrument nobody has calibrated. Teams live with it because people compensate. An agent doesn’t. The same honesty problem shows up when you score an agent trajectory and the suite is too kind.
The noise floor is higher than you think
Google published numbers on this years ago, and they’ve stuck with me, the same way a Jev tool-call risk score only helps if you know where the misses land. Across its whole corpus, about 1.5% of all test runs report a flaky result, meaning the same code both passes and fails. Almost 16% of its tests have some level of flakiness. And about 84% of the pass to fail transitions it observed involved a flaky test (Google Testing Blog ).
Read that last number again. When a test went from green to red, most of the time it wasn’t a real failure.
Google also did the math for a typical project. With about 1,000 tests and a 1.5% flaky rate, about 15 tests will likely fail on any given run, and each one needs someone to look at it. That’s at a company that takes testing more seriously than most. I’d bet your numbers aren’t better.
People compensate. Agents don’t.
A person on your team knows that one integration test fails every Tuesday. They rerun it and move on. That knowledge lives in heads and in old chat threads. It isn’t written down anywhere an agent can read.
An agent sees red and believes it. That goes one of two ways.
It chases a ghost. The agent rewrites working code to fix a failure that was never real. Now you have a diff nobody asked for, and a reviewer who has to figure out why.
It learns to retry. The agent notices that rerunning turns red into green. Soon it treats every failure as noise, including the real ones.
The second one is worse, because it looks like success. In the on-call loop from my last post , the agent’s fix PR is only as trustworthy as the tests that passed on it. If the tests lie, the fix is a guess with a green checkmark.
What agent ready actually means
“Agent ready” sounds like a new thing. It isn’t. It’s the stuff we always said we’d get to.
- One command builds it from a clean checkout. No tribal setup steps.
- One command runs the tests, and the result is the same every time on the same code.
- Lint and type checks fail loudly in CI. Warnings nobody reads don’t count.
- The repo says how to run, test and deploy, in a file in the repo, not on a wiki page from 2022.
- Tests are fast. If the suite takes 40 minutes, every agent attempt takes 40 minutes.
- Boundaries are clear. An agent changing one module shouldn’t need to understand all of them.
The 16-point self-audit
Score each line 0, 1 or 2. Zero is no, one is sort of, two is yes and you’d bet on it.
- A new laptop can build the repo with one documented command.
- The full test suite gives the same result on the same commit, five runs in a row.
- You know your flaky test rate, as a number.
- Flaky tests are quarantined with an owner and a date.
- Lint and type checks block merges.
- The suite runs in under 10 minutes.
- Run, test and deploy steps are written down in the repo.
- Dependencies are pinned and updated on purpose.
13 to 16, you’re ready to give an agent real work. 8 to 12, fix the gaps first and start the agent on small tasks. Under 8, start with the noise floor below. Anything you build on top of a noisy suite inherits the noise.
Fix the noise floor first
This is the order I’d go in.
Measure it. Run the suite five or ten times on the same commit. Any test that both passes and fails with no code change is flaky. That’s the same definition Google uses.
Quarantine, don’t delete. Move flaky tests to a non-blocking job. Give each one an owner and a date. A deleted test is lost coverage. A quarantined test is a ticket.
Kill the usual suspects. Tests that depend on the wall clock. Sleeps instead of waits. Calls to real network services. Shared state between tests. Tests that only pass in a certain order.
Pin your dependencies. A test that passed yesterday and fails today with no code change is often a dependency that moved.
Track it like an SLO. Put the flaky rate on a dashboard. If it goes up, treat it like any other reliability problem. It is one.
The payoff
Here’s the part I like. Every item on that list makes the codebase better for people too.
A new hire hits the same walls an agent does: setup steps nobody wrote down, tests that fail for no reason, a suite too slow to run before lunch. I wrote about that in maximizing developer onboarding , and the list barely changed. It also shows up in your DORA metrics . A suite you trust is how you ship more often without breaking more things.
So you were going to need this work anyway. Agents just made the bill come due.
Calibrate the instrument before you trust the reading. That was true on the bench, it’s true for your engineers, and it’s true for every agent you hand a repo to.