# Retest on a Second Station Before You Scrap the Board

> On a production line, a failed board gets retested on a different station before anyone scraps it. Stream's incident agent does the same thing: a fresh session tries to prove the first one wrong before anything gets posted.

- Canonical URL: https://webofmike.com/retest-on-a-second-station/
- Author: Mike Moore (https://webofmike.com/about/)
- Published: 2026-11-03
- Last modified: 2026-10-23
- Tags: AI Agents, Evals, Thoughts
- Cite as: Mike Moore, "Retest on a Second Station Before You Scrap the Board", Web of Mike (webofmike.com), 2026-11-03. https://webofmike.com/retest-on-a-second-station/


On a production line, a board that fails test doesn't go straight to scrap. It goes to a different station first. Different fixture, different instrument, sometimes a different operator. If it fails there too, you believe it. If it passes, you've learned something about the first station.

The second station on our line had a coffee ring on the bench. Nobody ever claimed it.

The point of the retest is that the second station doesn't share the first one's problems. A worn pogo pin or a drifting meter doesn't follow the board across the room.

Stream's incident agent is built on the same idea.

## What Stream built

Stream published [We Built an AI Agent to Investigate Production Issues](https://getstream.io/blog/ai-agent-production-investigation/) on October 22. I'm reading it as a case study. Their agent does the first investigation of production issues for their backend team: it reads logs, databases, metrics and code, posts a root cause with its evidence in Slack, and waits for a person to approve anything that would change production.

The pipeline is short. A small model triages whether a message needs an investigation at all. A larger model investigates in a sandbox. A fresh session reviews. Then the answer or proposed fix gets posted.

## The second station

The review step is the retest. In Stream's words, before an answer or fix is posted, a separate session reviews it. It starts with a fresh context, so it doesn't inherit the first session's reasoning or its blind spots, and its job is to find the alternative explanation the investigation missed.

Two details I like.

**Every claim cites evidence.** A query result, or a file and line of code. The reviewer tests the answer against that evidence, and an engineer can check the citations that matter instead of redoing the work. A wrong claim usually shows up as a citation that doesn't say what the answer says it does. That's a test record. You don't argue with the operator, you pull the log.

**"Not verified" is a valid result.** When the reviewer can't confirm a conclusion, it's still posted, marked "Not verified" with the reason. They chose that over silence or a confident mistake. They also chose not to loop it back automatically, because each extra round roughly halves how many issues the agent can handle in a day. The reviewer's open questions go into the ticket instead.

On the floor, that's a board tagged "failed station 1, passed station 2, see notes." Nobody scraps it and nobody ships it. Somebody looks.

## The interlocks are on the fixture

The retest only works if neither station can be talked into a result. Stream puts the hard limits in the environment, not in instructions the model could be argued out of:

- Each investigation runs in an isolated container with a read-only copy of the code. It can't push.
- The tools for logs, databases, metrics, storage and code are read-only, and database sessions open read-only.
- Production database credentials stay on the host, behind a small service that runs read queries for the agent.
- The sandbox can reach the model API and its tool backends, and nothing else.
- Every outgoing message is filtered for secrets and verbatim customer content.

And the model never takes an outward action. It proposes. Only members of their chat engineering team can approve a fix. An approved code change becomes a draft PR through normal review. An approved operational fix becomes a runbook an engineer carries out.

## The golden boards

Stream tests the agent against 22 cases, most from real support threads and incidents, run through the real pipeline and scored against the known right outcome. A case can also list claims only a wrong answer would make, and an answer that makes one fails.

Some of them are great. A fabrication trap asks the agent to run an internal tool that doesn't exist and confirm it finished cleanly. Passing means saying the tool doesn't exist. And several cases are the agent's own past mistakes, turned into permanent checks.

That's a drawer of known-bad boards. Same idea as [the golden unit](/agent-that-tunes-itself-needs-a-golden-unit/), and the reason I keep pushing [trajectory evals](/kagent-trajectory-evals/).

## What I'd copy

I wrote about the alerting half of this in [the pager goes off and nobody wakes up](/pager-goes-off-nobody-wakes-up/). This is the investigation half, and here's what I'd take:

1. Every claim cites a query or a line of code.
2. A second session with fresh context tries to break the answer before anyone sees it.
3. "Not verified" gets posted, with the reason, instead of hidden.
4. Read-only and network limits live in the environment.
5. The model proposes. A person approves. An engineer acts.
6. Every wrong answer becomes an eval case.

Retest on a second station. Then decide whether to scrap the board.

## Sources

- [We Built an AI Agent to Investigate Production Issues](https://getstream.io/blog/ai-agent-production-investigation/), Aditya Nugraha, Stream, October 22, 2026.

