kubectl is lying to you. Agent Substrate keeps the workload out of pods, so kubectl get pods shows a quiet WorkerPool while agents restore from snapshots, run a turn, and checkpoint back to object storage. This is the written version of my Nov 9 talk, and the hub for the four lessons that watching it turned up.
I built Substrate Scope because I was tired of explaining that gap with a whiteboard. Once people could watch chips move from storage to a worker bay and back, they got the runtime in a minute. Watching it also showed me things the CLI never would.
Why is kubectl lying to you on Agent Substrate?
Agent Substrate runs agents as snapshot-backed actors in gVisor sandboxes instead of always-on pods. Idle agents are snapshots in object storage. When a request comes in, the agent is restored onto one of a few pre-warmed workers, runs its turn, and checkpoints back out. One actor per worker, always.
That is a real density design. It is also invisible to the tool everybody reaches for. kubectl get pods shows the workers, a few MiB each, looking bored. It does not show the restore queue, the snapshot shelf, or which agent is in which bay right now.
Pods are the hardware. Actors are the workload. If your dashboards and autoscalers look at pods, they are watching the bench and not the board. The shorter how-to I want searchers to land on is kubectl is lying to you: watch Agent Substrate . The Monday teaser is kubectl Is Lying to You (Monday’s Talk) .
If you do not want my UI, read the same data the kagent controller already exposes:
kubectl port-forward -n kagent svc/kagent-controller 8083:8083
curl -s http://127.0.0.1:8083/api/substrate/status
Four lists come back: worker pools, actor templates, actors, and workers. Use the workers list. Lesson one is why.
What did the laptop cluster actually run?
From the lab, all of it in themsquared/kagent-substrate-demo :
- at least 18 agents on two worker pods
- over a thousand real chat sessions served
- minimum reserved capacity about 4.5x lower than the same agents as always-on pods
The earlier published lab ran nine agents. The talk setup doubles that on the same two workers. What is Agent Substrate is the one-page version of the runtime. The full build, including the gotchas with verbatim errors, is in the original lab post .
The comparison is reservation, not usage, on purpose. Idle always-on pods barely use CPU, but you pay for their requests around the clock. Each of kagent’s default always-on agents held about 204 MiB and requested 50m CPU while doing nothing in my cluster. Nine of those is roughly 1.8 GiB resident and 450m CPU reserved. Suspended actors do not hold that reservation. Two quiet worker pods look like nothing is happening. They held the live work. The rest of the fleet was on the shelf.
Scope itself is Apache-2.0, zero dependencies, with simulated mode for when you do not have a cluster and live mode for when you do. Live buttons are real: they scale the WorkerPool, fire surge traffic, run a demand-driven autoscaler, and reset a wedged pool.
What four lessons did Substrate Scope teach?
Each one has its own post. This page is the index, not a substitute.
- The Actors List Lies. Trust the Workers List. Suspended actors accumulate without bound and the list returns partial results under load. Join assignments from the workers list, which is bounded by pool size and names the assigned agent directly.
- CPU Is the Wrong Signal to Autoscale Agents . Workers are slot-bound, idle at single-digit mCPU mid-session, and real demand lives in callers’ retry loops. A CPU HPA sleeps through a full pool.
- The Autoscaler Made It Worse . On a serial model backend, more workers piled onto one generation queue. The bottleneck moved downstream and every turn got slower.
- Ghost Actors Pin Your Workers . Killed sessions left actors holding slots while the pool looked idle. Reset is cheap because state lives in object storage.
The thread through all four: the runtime is honest, but the views most of us use are views of the wrong layer. That is also why agent runtimes have an autoscaler, not a scheduler . Scaling answers how many workers exist. It does not answer who gets a slot when the pool is full. Substrate rejects. The caller retries.
What should you instrument on Monday morning?
A checklist, not a platform rewrite:
- Put a view on the actual workload. Scope is one option. Anything that reads the workers list works.
- Count rejections at the caller. The string is
substrate worker pool has no free workers. That is your demand signal until the runtime exports one. - Know your model backend’s concurrency before you scale workers in front of it. Slot count is not token throughput.
- Have a reset path for stuck slots, and know that it is cheap because state lives in object storage.
- Compare reservation, not usage, when you tell anyone the density story.
- Keep a checkpoint question on the list: does Agent Substrate’s checkpoint survive a held file lock? . Density only works for state the sandbox can serialize.
The rest of the runtime, isolation through scheduling, lives in the Agent Substrate field guide . The code is already public: themsquared/substrate-scope and themsquared/kagent-substrate-demo .