# Every Correction Is a Failure Report > Every correction is a failure report: Houck counted 564 product fixes across 14 patterns. Log them, Pareto them, and write the top ones into the skill. - Canonical URL: https://webofmike.com/every-correction-is-a-failure-report/ - Author: Mike Moore (https://webofmike.com/about/) - Published: 2026-11-14 - Last modified: 2026-11-08 - Tags: AI Agents, Skills, Thoughts - Cite as: Mike Moore, "Every Correction Is a Failure Report", Web of Mike (webofmike.com), 2026-11-14. https://webofmike.com/every-correction-is-a-failure-report/ Every correction is a failure report. Michael Houck pulled three months of product corrections he'd given his agent, 564 of them in 14 patterns, and wrote the fixes into a skill the agent checks next time. That is a defect log and a Pareto chart: log the misses, count the buckets, and keep counting after you write the rule. The defect log binder on the test bench had a rubber duck sitting on it. Nobody remembered who put it there. Everybody explained their failures to it anyway. That binder was the most useful document in the building. Not because any single entry was interesting. Because after a few months you could count them. ## What did Houck actually count? Michael Houck runs a single company agent with a brain of context and a library of markdown skills. In his [November 1 essay](https://michaelhouck.co/essays/i-gave-my-agent-an-mba), "I Gave My Agent An MBA," he describes two rounds of sending that agent to school. Round one was design, on October 1. He says everything the agent designed looked like an AI designed it, so he had it study Linear and Airbnb: essays from their design teams, interviews, social posts, and hours walking every page for actual CSS choices on spacing, color, and type. That became a design playbook the agent checks every screen against. Round two was product sense, after the playbook still missed UX. His example: a signup flow where someone types their email to get a free playbook, and the next screen asks where to send the playbook. So he had the agent go back through every time he'd corrected it on product decisions over three months. His count: **564 corrections, with 14 patterns.** Then he sent it to study practitioners (Lenny Rachitsky, YC talks and Paul Graham's essays, First Round Review, Stratechery) and built a skill from it. Now, in his words, "when I correct it, the fix goes into the skill and it checks against it next time." He packaged the process as a free meta skill he calls AI MBA: pick a topic, study practitioner sources, build skills, test them on real work. He is clear it is not done. The same essay says it built him a pretty bad survey flow that morning. A skill is not a cure. It is a place the next correction can land. One caveat: those numbers are his, counted by his agent. I haven't seen the 14 patterns, and they aren't published in the post. Treat 564 and 14 as one practitioner's report, not a lab result. [One run is an anecdote](/one-run-is-an-anecdote/) applies to a self-count too. ## Why is a correction a defect log entry? Strip the MBA framing and this is the oldest quality tool there is. Every correction is a failure report. It says what the system did, what it should have done, and someone noticed. Most people throw these away the moment they type "no, do it this way." Houck kept three months of them. 564 entries into 14 buckets is a Pareto chart waiting to happen. On any test floor I've worked, a few failure modes account for most of the defects. Fix those first, then recount. The signup-flow miss is a good specimen because it is specific. The user already typed an email. The next screen asked where to send the file. That is not "the model is bad at product." That is a repeated class of miss: the agent did not use information it already had. If that class is one of the 14, it belongs in the skill as a check, not as a pep talk. Houck's design round is the other specimen. He did not prompt "make it look better." He made the agent collect first-party evidence, then check new screens against that playbook. The product round did the same thing with corrections plus practitioner writing. In both cases the artifact is a document the agent actually reads. That last part is load-bearing. [My house agent rereads its rulebook](/house-agent-rereads-its-rulebook/) every time it wakes. If the fix lives in a Slack thread, it does not exist at 06:00. If it lives in the rulebook, it costs tokens, so it has to earn its keep. Houck's loop, "the fix goes into the skill," is the same rule: write it where the next run will see it. Ask an AI a question, he says, and you get the average of everything it has read. Make it go read the practitioners first, in their own words, and you get closer to their judgment. I care less about the MBA branding than about that order: evidence, then a check, then a recount. ## How do you run this on your own agent? 1. **Log corrections as data.** Each one gets what the agent did, what you wanted, and where. A line in a file is enough. Houck had three months of product corrections to mine because they already existed in the agent's history. If you do not log them, you will invent the patterns from memory, and memory is a terrible Pareto chart. 2. **Bucket them.** Let the agent propose patterns, then you merge and rename. Fourteen is a reasonable size. Forty means the buckets are too narrow. Two means you are hiding the interesting splits. You are the one who accepts the names. The agent does not get to grade its own scrap. 3. **Sort by count.** Write the fix for the top three into the skill or rulebook the agent actually reads. Mine lives in [the house agent's rulebook](/house-agent-rereads-its-rulebook/), and it's already long, so every rule has to earn its tokens. 4. **Recount.** After a few weeks, check whether those patterns dropped. If a bucket doesn't move, the rule isn't working, or the agent isn't reading it. Step four is the one people skip. Writing the rule feels like the fix. The count going down is the fix. Keep the binder. The duck is optional. ## What does a correction-to-skill test plan look like? If you are going to copy Houck's loop, copy the measurement, not just the slogan. 1. **Pick one domain.** Product decisions, or design, or deploy steps. Mixing every "no, not like that" into one pile will give you mushy buckets. Houck split design and product on purpose. 2. **Define a correction.** A human said the output was wrong and said what to do instead. A silent edit you made yourself still counts if you would have said it. A preference with no failure ("make it punchier") is not a defect unless you can write the missed check. 3. **Capture four fields.** Timestamp, what it did, what you wanted, artifact (diff, screenshot, prompt). Without the artifact you cannot tell whether the next run actually changed. 4. **Cluster, then freeze.** Agent proposes names, you merge. Freeze the 10 to 15 pattern names before you start scoring weeks. If you rename every week, the counts are not comparable. 5. **Write the top three as checks, not essays.** "Do not ask for an email the user just typed" is a check. "Have product sense" is not. Houck's signup example is already in that form. 6. **Put the checks where the agent loads them.** Skill file, rulebook, harness instruction. If the file is not in the prompt cache path, you did not ship the fix. You wrote a memo. 7. **Score the next two weeks.** For each frozen pattern, count new corrections. The only success is a drop in that pattern, not a nicer skill document. Houck still got a bad survey flow after the skill existed. That belongs in the next count. 8. **Retire or rewrite dead rules.** If a pattern does not move, the check is vague, the agent is not reading it, or the bucket was wrong. Do not add a fifteenth rule on top. [Trajectory evals](/kagent-trajectory-evals/) can watch whether the agent even called the skill. [Tests lying to your agent](/tests-lying-to-your-agent/) is what happens if your only signal is "the skill file exists." This is also how I would treat eval failures. A failed [Jev calibration](/jev-calibration/) case is a defect report. A failed trajectory case is a defect report. If you fix the prompt and do not keep the case, you are throwing away the binder. Houck's AI MBA packaging is optional. The log is not. ## Related - Source: [I Gave My Agent An MBA](https://michaelhouck.co/essays/i-gave-my-agent-an-mba), Michael Houck, Nov 1, 2026 - [Your house agent pays to reread its rulebook](/house-agent-rereads-its-rulebook/) - [Regression tests for kagent agents with agentevals](/kagent-trajectory-evals/) - [Your tests are lying to your agent](/tests-lying-to-your-agent/) - [One run is an anecdote](/one-run-is-an-anecdote/) - [Is Jev actually calibrated?](/jev-calibration/)