Testing Open-Weight Models as an Agent's Cheap Tier

Disclosure: I work at Solo.io, which created agentgateway and sells Solo Enterprise for agentgateway. The demos I cite are public repositories.

Two open-weight models, served locally through Ollama and routed through agentgateway , scored against Claude Sonnet 5 on the same 60-case agent tool-call risk benchmark. One of them failed to produce a parseable answer on 55% of calls. The other answered reliably, at 15 to 20 times Claude’s latency. Repo: themsquared/open-weight-cheap-tier .

An earlier experiment of mine, confidence-routed-agents, built a router that calls a cheap model first and escalates to Claude Sonnet 5 only when confidence is low. That post used TypeSafe AI’s Jev as the cheap model, a typed-decision API rather than a chat model. The obvious next question: what if the cheap tier is an open-weight LLM you run yourself, with no per-call vendor charge at all? This is that test.

What was built

Three backends behind one agentgateway v1.6.0 instance, all answering the identical OpenAI-compatible /v1/chat/completions request:

• qwen3.8: 27.3B parameters, Q4_K_M quantization, 262K context, served by a local Ollama instance.
• nemotron-3.5-lightning: 32.9B parameters (sparse MoE, nemotron_h_moe architecture), Q4_K_M, 1M context, also served by Ollama.
• claude-sonnet-5: the frontier baseline, routed through agentgateway’s anthropic provider.

tasks.jsonl (60 cases, from jev-benchmark)
        |
        v
  agentgateway v1.6.0 /v1/chat/completions
        |
   +----+----+----------------+
   |         |                |
qwen3.8  nemotron-3.5-     claude-sonnet-5
(ollama) lightning (ollama)  (anthropic)

One gateway, one request shape, three backends. The task set is the 60-case hand-labelled agent tool-call risk classification set from jev-benchmark : classify a tool call as readonly, destructive, privileged, or exfiltration, split 34 clear / 14 ambiguous / 12 adversarial. Reusing it verbatim means these numbers sit directly next to jev-benchmark’s and confidence-routed-agents’ on the same scale.

A quick note on which models these actually are, because the field is loudly discussing a different one: HN this month has 929 points on running a 125B-parameter “Qwen3.8 Flash Next” on an RTX 4090. The qwen3.8 pulled from Ollama’s library and measured here is a 27.3B-parameter model that shares a name prefix and nothing else. GLM-5.3 Flash and DeepSeek V4.1 Flash, also part of that discussion, were not tested at all: no hosted API key for either on this machine, and both are too large to run locally on an Apple M4 Max without a further quantization pass this post didn’t attempt. Two real, locally-available candidates, reported honestly, beat four candidates where two are guesses.

The gotcha: the OpenAI-compatible path can’t turn off thinking

Both Ollama models are reasoning models by default. Ollama’s own documentation describes a "think": false parameter on its native /api/chat endpoint, and it works: a direct call with it returns a clean answer in well under a second.

agentgateway’s provider: ollama doesn’t speak that native endpoint. It speaks Ollama’s OpenAI-compatible /v1/chat/completions endpoint, matching the OpenAI chat schema. That endpoint silently ignores "think": false entirely, confirmed by sending it both through the gateway and straight to Ollama with the identical result: an empty content field, finish_reason: "length", and the full max_tokens budget spent on an unterminated reasoning field that OpenAI’s schema has no place for.

curl -s http://localhost:3310/v1/chat/completions -d '{
  "model": "qwen3.8", "think": false, "max_tokens": 20,
  "messages": [{"role":"user","content":"Reply with exactly: OK"}]
}'
# {"choices":[{"message":{"content":"","reasoning":"The user wants me to
#   reply with exactly \"OK\". This is a simple instruction - I just"}, ...
#   "finish_reason":"length"}], ...}

There is no parameter fix on this path. The only lever is max_tokens large enough for the model to finish reasoning and still emit the answer: this benchmark used 700 for qwen3.8 and 900 for nemotron-3.5-lightning. That still was not enough on every case, which is most of this post.

Results

n=60 per model, one run, 2026-10-09:

Accuracy of all 60 counts an unanswered case (empty content, a parse failure) as wrong, the same way a production router would have to: a caller that gets back HTTP 200 and an empty string has no decision to route on, same as if it had gotten the wrong one.

nemotron-3.5-lightning is the headline. When it answers, it is perfect: 27 for 27, and noticeably faster than qwen3.8 (median 6.5s vs 18.9s). But it only answers 45% of the time. On the other 55% of calls, it spent its entire 900-token budget mid-reasoning and returned nothing usable. That is not a model that is wrong sometimes; it is a model that, through this interface, frequently doesn’t finish.

qwen3.8 is the more usable of the two: it answers 90% of the time, and is right on 98.1% of what it answers. But “cheap” has to count latency, not just dollars. A median of 18.9 seconds per call, with a p95 over 42 seconds, is 15 to 20 times slower than Claude Sonnet 5’s 1.08-second median on the identical task. A confidence-routed agent built to escalate the hard cases to a frontier model only makes sense if the cheap path is actually cheap on the dimension that matters for an interactive agent loop, and here the dimension that breaks first is time, not money.

By difficulty, for cases each model actually answered:

Every model gets the easy third right every time. nemotron-3.5-lightning’s unanswered cases concentrate on the harder slices (only 3 of 14 ambiguous and 3 of 12 adversarial cases got an answer at all), which tracks: harder cases need more reasoning tokens, and the budget ran out more often exactly where the task needed more of it.

What this does and does not show

This is one 60-case classification task on one Apple M4 Max, not a general open-weight-versus-frontier verdict. The $0 cost for the Ollama models is real for hardware that is already paid for and otherwise idle; it is not the same comparison as a cloud GPU instance billed by the hour, which this post doesn’t measure.

The failure mode here is specific to agentgateway’s Ollama provider, which speaks the OpenAI-compatible endpoint. Calling Ollama’s native API directly, with "think": false actually honored, would likely close most of the answer-rate gap. That is a different, ungoverned path: no gateway budget, no unified access log, no routing policy in front of it, which is the entire reason a gateway is there in a confidence-routed agent in the first place. Raising max_tokens further on the OpenAI-compatible path would also answer more cases, at the cost of worse p95 latency on exactly the hardest ones. Both are real engineering tradeoffs to make deliberately, not oversights to tune away before shipping.

Try it

git clone https://github.com/themsquared/open-weight-cheap-tier
cd open-weight-cheap-tier
ollama pull qwen3.8
ollama pull nemotron-3.5-lightning
export ANTHROPIC_API_KEY=sk-ant-...
docker compose up -d
python3 bench.py --model qwen3.8
python3 bench.py --model nemotron-3.5-lightning
python3 bench.py --model claude-sonnet-5

Full config, raw per-case results, and the benchmark harness are in the repo: themsquared/open-weight-cheap-tier .