I open-sourced Substrate Scope, a live visualizer for Agent Substrate, the Kubernetes-native runtime that runs AI agents as snapshot-backed actors in gVisor sandboxes instead of always-on pods. The code is at themsquared/substrate-scope, Apache-2.0, zero dependencies. You need node 18+ and kubectl.
Why a visualizer for this runtime
Agent Substrate’s whole point is that agent state does not live in pods. Idle agents are zstd snapshots in object storage. Active agents are actors that get restored onto a small pool of pre-warmed gVisor workers, run a session, and checkpoint back out. It’s a strong design, and I wrote about standing it up in my last post.
The problem is that kubectl get pods shows none of it. Two worker pods sitting at 14 MiB each looks like nothing is happening while nine agents serve hundreds of sessions through them. When I demoed this to people, the reaction changed completely once they could watch it: chips physically flying from storage to a worker bay, glowing while the LLM turn runs, and flying back as a checkpoint. The runtime is genuinely interesting to watch, and watching it is the fastest way to understand it.
What the board shows
The layout reads top to bottom, the same direction the architecture works:
- WorkerPool bays: each pre-warmed gVisor sandbox, which agent is running in it right now, and how many sessions it has served. One actor per sandbox, always.
- Restore queue: agents waiting when demand exceeds the pool. Substrate rejects new sessions when the pool is full rather than queuing them, so the retry queue lives client-side and the board renders it from client reports.
- Object storage shelf: every suspended agent as a snapshot card with its per-agent snapshot count.
- Telemetry: four charts, and the one I care most about compares reserved capacity. The dotted line is what the same agents would reserve as always-on pods (agents times per-pod requests, flat forever). The solid line is what the workerpool reserves (slots times the same unit), and it steps down as the autoscaler releases workers. Reservation is the honest comparison: idle pods use almost no CPU but you pay for their requests around the clock.
- Per-agent drawer: click any chip and get that agent’s stream. Prompts in, replies out with latency, errors verbatim, restores and checkpoints. Substrate exposes network ingress only into sandboxes, no exec and no logs, so chat I/O plus lifecycle is exactly what an agent’s observable output is.
The controls are real, not theater. The workers plus and minus buttons run kubectl scale workerpool against the live CR. AUTOSCALE turns on a demand-driven scaler (queue depth up, windowed idle capacity down, one jump to target in either direction). RESET POOL recovers workers wedged by aborted sessions. STOP DEMO is a master kill switch for anything that costs LLM tokens, which exists because I once left the load generator running overnight against a paid API.
How it gets live data
The server watches your current kubectl context and picks a source automatically:
- kagent source: if you run substrate through kagent (0.9.7+), Scope polls the controller’s substrate inventory endpoint and gets full fidelity: per-actor runtime state and worker assignments straight from ateapi, plus chats you send from the kagent UI ingested into the drawer.
- crd source: on any substrate cluster without kagent, Scope falls back to kubectl: WorkerPools, live worker pods, and ActorTemplates with their golden-snapshot phase. Scaling and autoscaling still work. What it can’t see is which actor is on which worker right now, because that state lives in ateapi behind gRPC and JWT auth.
Closing that gap is the first issue on the repo: a direct ateapi adapter. The upstream Control service already exposes ListActors and ListWorkers, so the work is gRPC client wiring plus ServiceAccount token auth, mapped onto the snapshot shape the server already emits. The frontend needs zero changes. If you want a well-scoped contribution to a young project, that’s it.
One implementation note that generalizes: derive worker assignments from the workers list, not the actors list. The actor inventory grows without bound (every completed session leaves a suspended actor entry) and under load it returns partial results. The workers list is never larger than your pool. I learned this by watching running sessions randomly vanish from the board until the join was rebuilt.
Quickstart
git clone https://github.com/themsquared/substrate-scope.git
cd substrate-scope
node server.mjs # simulated mode: no cluster needed, full animation
node server.mjs --live # live mode: watches your current kubectl context
Open http://localhost:8123. Simulated mode is the whole experience with synthetic data, which makes it useful for talks even without a cluster.
On kagent clusters, the included load generator drives real chats so the board moves:
node stimulate.mjs --budget 300 # stop after 300 chats, spend-capped
node stimulate.mjs --oversub 6 --load 0.9
Every chat is a genuine actor restore, an LLM turn, and a checkpoint, and each one shows up in the per-agent drawer with its latency.
Where this goes
Near-term roadmap, in rough order: the ateapi adapter above, surfacing actor logs through substrate’s kubectl-ate logs actors path, and multi-pool support for clusters running more than one WorkerPool. Issues and PRs welcome at themsquared/substrate-scope.
If you want the full story of standing up Agent Substrate with kagent, including every gotcha with verbatim error messages, that’s in the previous post. This tool exists because that work convinced me the runtime deserves to be seen, not just described.