# Confidence-Routed Agents: Cheap First, Frontier on Escalation

> A router calls Jev for agent tool-call risk, escalating to Claude via agentgateway only on low confidence. Escalating everything cost more and scored worse.

- Canonical URL: https://webofmike.com/confidence-routed-agents/
- Author: Mike Moore (https://webofmike.com/about/)
- Published: 2026-11-02
- Last modified: 2026-11-02
- Tags: AI Agents, AI Gateways, Security, Generative AI
- Cite as: Mike Moore, "Confidence-Routed Agents: Cheap First, Frontier on Escalation", Web of Mike (webofmike.com), 2026-11-02. https://webofmike.com/confidence-routed-agents/


Escalating every uncertain agent decision to a frontier model sounds like the safe default. On a 240-case benchmark it was also the worse one: routing by confidence and escalating only when a cheap model said it wasn't sure cost half as much as escalating everything, and scored higher.

[jev-benchmark](https://webofmike.com/jev-benchmark/) measured [TypeSafe AI's Jev](https://typesafe.ai) against Claude on the same agent tool-call risk task. [jev-calibration](https://webofmike.com/jev-calibration/) went further and measured whether Jev's own confidence score could be trusted to make a routing decision, and found a specific, usable threshold: a confidence of exactly 1.000 was right 133 times out of 134 on 240 labelled cases. Neither post built the thing that threshold is for. "Escalate when confidence is low" is not a model feature, it is a two-call routing problem, and routing problems belong at a gateway. This is that gateway, built and run for real: [themsquared/confidence-routed-agents](https://github.com/themsquared/confidence-routed-agents).

## What was built

A router calls Jev directly for every task (Jev is a typed-decision API, not a chat model, so it isn't an LLM backend agentgateway knows how to proxy). When Jev's confidence clears the threshold, its answer is the answer. When it doesn't, the router escalates to Claude Sonnet 5, and that one call goes through [agentgateway](https://agentgateway.dev) v1.5.0 instead of straight to Anthropic.

```
            ┌──────────────┐  confidence >= threshold   ┌─────────────┐
 task ──────▶   router.py   ├───────────────────────────▶  answer     │
            │  (calls Jev   │                            │  (cheap     │
            │   directly)   │  confidence < threshold    │   path)     │
            └──────┬────────┘                            └─────────────┘
                   │
                   │ escalate
                   ▼
          ┌─────────────────────┐   budget OK    ┌───────────────┐
          │   agentgateway       ├───────────────▶ Anthropic API  │
          │  (holds the real     │   budget       │ claude-sonnet-5│
          │   Anthropic key,     │   exceeded     └───────────────┘
          │   per-key USD budget,│   ──▶ HTTP 429
          │   access log)        │
          └─────────────────────┘
```

Why the gateway sits only on the escalation half: that's the half with a shared, billable credential. Jev's key is the router's own, scoped to a typed-decision API with no spend risk from a runaway loop in the same way a chat model has. The escalation key is exactly the kind of credential this campaign has argued belongs behind a gateway since [secretless-ai-agents](https://webofmike.com/secretless-ai-agents/): the router never holds it, agentgateway injects the real `x-api-key` on the way out, and the budget and the audit trail live in one place instead of scattered across whatever called the model.

The task set is the same 240 hand-labelled agent tool calls from [jev-calibration](https://github.com/themsquared/jev-calibration): classify each one as `readonly`, `destructive`, `privileged`, or `exfiltration`. Reusing it instead of sampling a new set means these numbers are comparable to that post's, not just another unrelated benchmark.

## The result that mattered

Six configurations against the identical 240 tasks, one live Claude Sonnet 5 call per escalation:

| Threshold | Escalated | Accuracy | Total cost | Avg cost/call |
|---|---|---|---|---|
| 0.00 (Jev only, never escalate) | 0/240 (0%) | 90.0% | $0.0042 | $0.0000174 |
| 0.70 | 32/240 (13.3%) | 92.9% | $0.0262 | $0.0001090 |
| 0.85 | 41/240 (17.1%) | 92.1% | $0.0356 | $0.0001484 |
| 0.95 | 61/240 (25.4%) | 92.9% | $0.0520 | $0.0002168 |
| 0.99 | 83/240 (34.6%) | 93.3% | $0.0646 | $0.0002692 |
| 1.00 (jev-calibration's own cutoff) | 110/240 (45.8%) | 93.3% | $0.0846 | $0.0003523 |
| Never use Jev (all escalate) | 60/60 on a 60-case subset | 91.7% (55/60) | $0.0422 (60 calls) | $0.0007035 |

Two things here are worth stating plainly instead of smoothing over. The 0.85 row scores a hair worse than both neighbors (92.1% against 92.9% on either side), which at n=240 is noise from a handful of cases crossing the cutoff, not a real dip. Read the 0.70-0.95 band as roughly 92-93%, not as a precise function of the threshold.

The real finding is the last two rows against the bottom one. Routing at jev-calibration's own recommended cutoff, escalate anything below exactly 1.000 confidence, cost half of what escalating everything costs ($0.0846 against an extrapolated $0.169 for 240 all-Claude calls) and scored better, not worse: 93.3% against 91.7%. The cheap path is not a compromise that trades accuracy for savings here. Jev's answers at exactly 1.000 confidence were already shown, in the prior post, to be right 133 times out of 134. Sending one of those cases to Claude anyway does not improve on that; Claude's terse, non-reasoning JSON classification call turned out to be the less reliable of the two on the cases Jev was already sure about. Escalating them just spends money confirming an answer you already had right.

Going from 0.99 to 1.00 is its own small lesson: it adds 27 more escalations and about $0.02 more cost for zero accuracy change (93.3% both). Past a certain point, a tighter threshold just buys more spend, not more correctness. The threshold that is "tight enough" is a measured fact on a specific task set, not a number to default to 1.0 on faith.

## The budget actually blocks

A threshold set wrong, or an upstream regression that makes Jev return uncertain answers more often, turns an "occasional escalation" path into "most requests hit the expensive model." That is exactly the failure mode a gateway budget exists to cap. Verified with a temporary $0.0002/24h budget, not the real $1.00 one that ships in the repo's config, to prove the mechanism without burning hundreds of calls:

```
call 1 -> HTTP 200
call 2 -> HTTP 200
call 3 -> HTTP 200
call 4 -> HTTP 429
{"error":{"message":"Budget exceeded","type":"rate_limit_error","code":"budget_exceeded"}}
```

agentgateway's own log line for the rejected call:

```
2026-10-01T14:39:20.199122Z  warn  budget  API key budget exceeded  connection.id=4 request.id=4 api_key="router" budget="daily-usd" used=0.000216000 limit_unit="USD" limit_amount=0.0002 exceeded=true
```

`used` and `limit_amount` are both computed from the three real calls that preceded the rejection, priced from agentgateway's own `modelCatalog` rates for `claude-sonnet-5`, not round numbers chosen to make the demo work.

## Jev being unreachable does not stop the agent

The router's `--no-jev` flag simulates the cheap path being down: every task falls through to the Claude escalation path with no special-casing in the code, because "escalate" is already the router's only other branch. A production version of this would want the inverse check too (what happens if the frontier path is the one that's down), but the asymmetry is deliberate here: Jev being unreachable should degrade to "more expensive, still correct," not "the agent stops answering."

## A gotcha worth keeping

One case in an earlier full-240 run came back from Claude with no `content` block at all, just a `finish_reason` and nothing else, at `max_tokens: 128`. This is the identical failure the jev-calibration harness already hit at the same token ceiling: Claude's internal reasoning consumed the whole response budget before producing the JSON answer. Raising the cap to 256 did not reproduce it across all 240 cases, and `router.py` checks for a missing or empty `content` field explicitly rather than assuming the key is always there. A single silently-dropped answer in an agent pipeline is worse than a slow one; code that assumes the happy-path response shape will eventually hit this.

## What this does not cover

No frontier-vs-frontier comparison; Claude Sonnet 5 is the only escalation backend tested here. No measurement of sustained escalation load over days, only a point-in-time budget trip. And the $1.00/24h budget shipped in the repo's config is a demo value, not a sizing recommendation, you'd size that from measured escalation rate and real model pricing for your own workload.

The harness, the task set, the raw per-call results at every threshold, and the agentgateway config are in [themsquared/confidence-routed-agents](https://github.com/themsquared/confidence-routed-agents).