Run kubectl get pods on an Agent Substrate
cluster under load and you’ll see two worker pods, calm, a few MiB each. Nothing to see here.
Meanwhile agents are restoring from snapshots, waiting for a slot, running a turn, and checkpointing back out to object storage. None of that is a pod. None of it shows up.
On Monday, Nov 9, I’m giving a talk about that gap: kubectl Is Lying to You: What a Live Visualizer Taught Me About Agent Substrate.
What’s in it
I built Substrate Scope , an open source live visualizer, because I got tired of explaining the runtime with a whiteboard. Once people could watch agents fly from storage to a worker bay and back, they got it in a minute. Watching it also taught me things I would not have found from the CLI.
The setup: at least 18 agents on two worker pods, over a thousand real sessions, and reserved capacity about 4.5x lower than the same agents as always-on pods. All on a laptop, all open source. (The first lab writeup ran nine; the talk doubles it.)
The four lessons, each getting its own post after the talk:
- The actors list lies. Suspended actors pile up and the list returns partial results. The workers list is the source of truth.
- CPU and memory are the wrong autoscaling signals. Workers are slot-bound and idle at single-digit mCPU, and demand lives in your callers’ retry loops.
- The autoscaler made it worse. On a serial model backend, more workers just piled onto one generation queue.
- Killed sessions leave ghost actors. They pin workers while the pool looks empty.
If you’re coming, say hi. If you’re not, the recap and all four lessons land here next week, and the code is already public: themsquared/substrate-scope and themsquared/kagent-substrate-demo .