# An Agent That Tunes Itself Needs a Golden Unit > You never trust a test station that recalibrated itself until it measures a known-good unit. Agents that rewrite their own prompts need the same thing. Kiro's self-fix loop shows what that looks like. - Canonical URL: https://webofmike.com/agent-that-tunes-itself-needs-a-golden-unit/ - Author: Mike Moore (https://webofmike.com/about/) - Published: 2026-11-01 - Last modified: 2026-10-21 - Tags: AI Agents, Evals, Thoughts - Cite as: Mike Moore, "An Agent That Tunes Itself Needs a Golden Unit", Web of Mike (webofmike.com), 2026-11-01. https://webofmike.com/agent-that-tunes-itself-needs-a-golden-unit/ On the test floor there's one rule about calibration that nobody argues with. When a station has been adjusted, you don't trust it until it measures the golden unit. A known-good device, kept in a drawer, with known readings. Run it through. If the numbers match, the station goes back on the line. If they don't, it doesn't matter how confident the calibration routine was. Ours lived in an anti-static bag with a strip of masking tape on it that said "Tuesday." The station that calibrates itself and then reports "calibration passed" is the one you watch closest. It has every reason to believe itself. Agents that rewrite their own prompts and tools are that station. ## What Kiro built The Kiro team published [How Kiro fixes itself, and how we keep the fixes honest](https://kiro.dev/blog/how-kiro-fixes-itself/). They built an internal system called kiro-trace that reads Kiro's own runs, finds where it fails or wastes effort, proposes a change, and keeps the change only if it survives checks built to reject it. In its first week, [per the post](https://kiro.dev/blog/how-kiro-fixes-itself/), kiro-trace opened eight pull requests. Seven merged, one was closed for unrelated reasons, and none were sent back in code review. All seven changed prompts or tools: tool descriptions, argument checks, response behavior. They're clear that the numbers come from Kiro's internal usage at Amazon, and each figure is a delta from an internal experiment, not a production estimate. Read it as a case study of the method. ## The near miss The best part of the post is a fix they almost shipped. It passed more tasks and used fewer tokens. Then they split the results by host and found almost all the gain came from one run where the agent happened to escape a loop it usually gets stuck in. Without that host, the fix was slower, more expensive and less accurate. Their line: "A loop that grades its own fixes would have shipped it." Every test engineer has a version of this story. The yield jumped on second shift, everybody celebrated, and it turned out one station's limits had been loosened. The number was real. The improvement wasn't. ## Where the golden unit shows up Kiro doesn't use the phrase, but the parts are all there. **The judge is not the proposer.** They keep three roles separate: Kiro is the agent being improved, the loop proposes and applies a change, and a gate made of plain code decides whether it survives. A human makes the final merge call. They say they learned this partly from their own mistake, reading the same run three different ways while judging by hand. The calibration routine does not get to sign off on its own calibration. **A fresh control every time.** They measure the delta, not the score, and re-run the unchanged control on every comparison, because a new model alone can move the score. That's measuring the golden unit on the same day, on the same station, not comparing against last month's printout. **Regression on what already passes.** Candidates also run on tasks Kiro already passes, and the final candidate is compared on the full benchmark against the unchanged control. A change has to show it did not regress against a set margin. A calibration that fixes channel 3 and breaks channel 7 fails. **A suite that only grows.** Every confirmed fix adds a deterministic assertion to a permanent regression suite, so a failure fixed once doesn't come back without being caught. Every check in the loop exists because the loop or the team was fooled once. That's how a golden unit set grows on a real floor: every escaped defect earns a new known-bad sample in the drawer. **A contamination check.** Any pass where the agent used the network to look up a task's answer gets thrown out. On the floor, that's checking nobody taped the expected readings to the side of the fixture. ## What the fixes looked like Some of them are almost funny in how small they are. A false claim about harness behavior in the file-read and search tool descriptions reached the model 61,742 times across 445 samples, and they removed it. Guidance told the model to call a tool that does not exist, and they deleted it. Redundant file re-reads dropped about 83 percent. A silent file-search bug was breaking roughly one in eight searches. The headline result was compaction. Kiro could abandon its current task after a compaction and resume a much older one, because the summary ended on the oldest user message. The fix removed that raw message list, and their retention score went from [0.82 to 0.92](https://kiro.dev/blog/how-kiro-fixes-itself/). Notice the pattern. They say the loop tries removing something before adding a rule by default. Most of the wins were deletions. The biggest change in the post has not shipped. A smaller tool and prompt profile scored 82.4 percent against 77.5 percent pass@1 on Terminal-Bench 2.1 at roughly a third of the token cost, and the team is still deciding whether to make it the default, because it would change every user's tools and prompt. That's the right call. A benchmark win is a station passing the golden unit. Changing the default for everyone is a product decision. ## The checklist If your agent tunes itself, or you're letting another agent tune it, here's what I'd want before trusting the output: 1. The thing that proposes a change cannot be the thing that accepts it. 2. Acceptance lives in code the model cannot reach. 3. Every comparison re-runs an unchanged control. 4. Every change runs against tasks that already pass, with a margin, not just a "no significant difference." 5. Every confirmed fix adds a permanent check. The suite never shrinks. 6. A human signs the merge. I made the same argument from the other side in [your tests are lying to your agent](/tests-lying-to-your-agent/) and in [who signs the robot test report](/who-signs-the-robot-test-report/). The checks are only worth something if the thing being checked can't edit them. Recalibrate as often as you want. Then measure the golden unit.