Agent action verification without asking the LLM if it worked

Agent action verification should be a binary check on the call path, not a second large-model prompt that says it is 90 percent sure. Evaluate a yes/no condition on the tool call or its result. Do not ask the same LLM that just ran the tool to inspect the terminal output and narrate confidence.

I have watched that pattern in demos. The agent cats a file, then spends a full generation confirming the cat. You pay latency and you get a sentence like “I am 90% sure.” That sentence is not a control. On Substrate the waste is worse. The actor is holding a worker slot while it waits on that second generation. Verification that needs a frontier model is a capacity bug, not only a safety bug.

Why is asking the LLM if it worked unsafe?

The model writes the number token by token like any other text. It is a plausible continuation of the conversation, not a measurement of anything. Ask again and it can change, for example 0.85 one time and 0.7 the next. Rewording the question, like asking “Are you sure?”, usually moves it too. The words before the number decide the number. If the model already wrote “This is clearly the right answer,” a high score is the natural next thing to write.

I wrote that up as an LLM’s confidence score is a meter with no cal sticker . On a test floor, every meter has a sticker. It says who calibrated it, against what, and when it is due again. A meter without one is not an instrument. It is a box that shows numbers.

The rumple post the cal-sticker piece walks through is the same rule: do not ask the device under test to grade itself. Build a measurement from things you can observe, then keep checking it against outcomes. If nobody records whether answers turned out right, there is no sticker.

A second generation also burns a worker slot. Substrate workers are slot-bound. One actor per worker. An LLM turn is mostly I/O wait on the model provider. Workers hosting active sessions measured at single-digit mCPU, which means the bay is still taken while the “did that work?” prompt sits in a queue. I have already seen extra workers make a serial generator worse. Adding a verification round trip to every tool call is another way to fill the same cashier line. The autoscaler made it worse .

Positional bias in long prompts is a reason not to stuff the whole transcript into a second LLM judge. A small model with a typed input (tool name, args, a short result) is the point. A judge that reads 40k tokens of agent rambling will agree with the last paragraph.

What belongs on the call path instead?

A boolean. Allowed or not. Succeeded or not.

Deterministic checks first: exit code, JSON schema, HTTP status, “did the file exist after the write.” Free, fast, boring. Use them. Exit code 0 after mkdir does not need inference.

When the check is semantic (“is this kubectl destructive?”), I want a binary decision model with a confidence I have actually calibrated, not a chat completion. I ran that experiment on Jev: 60-case tool-call risk benchmark and calibration on 240 cases . The task is one typed decision: classify an agent tool call as readonly, destructive, privileged, or exfiltration.

The operational finding was not “always right.” It was: treat 1.000 as a different kind of number than 0.97, and never route on a score you have not graded. On 240 hand-labelled calls, a confidence of exactly 1.000 was right 133 times out of 134. Scores between 0.90 and 0.99 averaged 0.970 and were right only 87.7% of the time. The headline ECE of 0.057 averages both of those into one small number. On the 60-case set, accuracy was 91.7% (55/60), and every incorrect answer came with confidence below 1.000.

I do not need 15ms in this post. I need “fast enough that I will actually leave it inline.” Jev’s p50 on the 240-case set was 267 ms, p95 349 ms. That is not tens of milliseconds. It is also not a second-plus generation. I would still leave it inline. If the check is 1.5 seconds, someone will skip it under load.

Binary verification for agents is the calibration how-to. The interface is the whole point: tool name, arguments, maybe a short result in, a class or boolean plus confidence in [0, 1] out. If the library cannot return that shape, it is another chat model. I will not put it on a deny path.

Do not confuse this with identity. Verification answers “may this action happen.” Identity answers “who is asking.” You need both. Proof of agent identity . A calibrated no from nobody in particular is theater.

Where does the check sit on Agent Substrate?

The actor is the wrong place to keep the policy model, for the same reason it is the wrong place to keep secrets: it suspends, restores, and should not be the policy authority. Idle agents are snapshots in object storage. A request restores an actor onto a free worker. The worker’s ServiceAccount is the pool, not the agent. Policy that lives in the actor suspends with the actor and becomes optional.

I want:

1. Actor emits a tool call.
2. Gateway or sandbox hook evaluates the boolean.
3. Deny stops the call. The actor never sees a forged success.
4. Allow records the decision somewhere the agent cannot write.

That is the same chokepoint as tool-call guardrails . agentgateway can reject or rewrite an MCP tools/call before the tool runs, from a request-phase processor. A request-phase rejection means the tool is never invoked. A response-phase processor can only transform or pass through, because the tool has already run. Put verification on the request path when the question is “may this happen.” Put a result check on the response path when the question is “did it succeed,” and do not confuse the two.

Ephemeral actors help here. There is no long-lived sidecar you forgot to update. The hook is on the platform. Every restore picks it up. Halt the action before the next tool runs. A denied call must not look like a missing tool if you can help it. Agents retry missing tools. They should stop on deny.

Unattended automation raises the bar. No human in the loop means no “are you sure” and a much higher bar on this check. Agent platform at scale .

What tradeoffs will actually turn the guardrail off?

False deny vs false allow. For anything that spends money, deletes, or leaves the cluster, I fail closed. For “summarize this log,” I fail open and log. If you use one threshold for both, you will turn the guardrail off.

The 0.9 band is where this bites. On the 240-case set, a threshold of 0.9 would have let through eight confident misses, including a grep classified as destructive and three credential reads classified as readonly. Those credential reads were confident because my criteria text called them reads. Calibration is measured against your criteria, not your intentions. If credential reads matter, name them in the criteria.

Speed is the other off-switch. If I cannot run it on every call, I will sample, and sampling is how the one refund sneaks through. Optimize the check. Do not sample money movement.

A second LLM judge also fights the pool. While the actor waits, it holds a worker. Eighteen agents on two workers only works if idle time is real. A verification generation on every tool call is how you fill the bays with “I am 90% sure.” Who runs next when the agent pool is full .

Checklist: put a boolean on the tool-call path

1. Write the deterministic checks first: exit code, schema, HTTP status, file exists after write.
2. For semantic risk, use a typed yes/no model. Input is tool name, args, short result. Output is a class plus confidence in [0, 1].
3. Grade the confidence on your own labels before you threshold. Treat 1.000 and 0.97 as different objects if your curve says so.
4. Fail closed for spend, delete, and egress. Fail open and log for low-impact reads.
5. Run the check at the gateway or sandbox hook, not inside the actor. Deny must not look like a missing tool.
6. Record every decision with an input hash so you can regrade next month.
7. Do not add a second large-model “did that work?” turn. It is slow, uncalibrated, and it holds a worker slot.

Field guide: Agent Substrate . Trajectory scoring on sessions is a sibling, not a substitute: kagent trajectory evals .