Agent Substrate is a Kubernetes-native runtime that breaks the pod-per-agent model. Instead of one always-on pod per AI agent, idle agents are checkpointed to object storage as compressed snapshots and restored on demand into a small pool of pre-warmed gVisor sandboxes. Agent count stops being a pod count: your fleet lives in object storage at near-zero marginal cost, and the design point is thousands of agents multiplexed across tens of pods. The kagent UI’s own source comments size its substrate inventory page for “a cluster running four hundred thousand” actors, which tells you where the project thinks this goes.

I proved the mechanics end to end on a laptop: a kind cluster with kagent, nine agents on two worker pods, over a thousand real chat sessions served, reserved capacity 4.5x lower than the same agents as always-on pods, and a live visualizer that shows every restore, queue wait, and checkpoint as it happens. The whole thing, visualizer included, is in themsquared/kagent-substrate-demo on GitHub. The ratio is the same machine at any scale; only the numbers get bigger.

What Agent Substrate is

Always-on agent pods waste capacity because agents are idle most of the time. On my cluster, each of kagent’s default always-on agents holds about 204 MiB of memory and requests 50m CPU while doing nothing. Nine of those is roughly 1.8 GiB resident and 450m CPU reserved around the clock.

Agent Substrate decouples the agent’s lifecycle from pod infrastructure:

  1. When an agent is invoked, its actor is restored onto a free worker from a WorkerPool, rehydrated from a zstd snapshot in object storage.
  2. The agent runs inside a gVisor sandbox for the duration of the session. One actor per sandbox, always.
  3. When the session ends, the actor’s state is checkpointed back to object storage and the worker slot frees for the next agent.

A worker hosts one actor at a time. The density comes from time-multiplexing: worker-0 in my cluster served 50+ sessions across nine different agents in an afternoon. Simultaneous sessions equal your worker count; total agents are limited only by object storage.

The runtime is two CRDs (WorkerPool and ActorTemplate in the ate.dev group) plus a control plane (ateapi, backed by a Valkey cluster), a data-plane router (atenet), a per-node snapshot mover (atelet), and a worker supervisor (ateom) that talks to gVisor’s runsc. A detail I enjoyed: atenet is agentgateway under the hood. Its config lives at /etc/agentgateway/config.yaml inside the pod.

What the visualizer shows

kubectl get pods makes substrate look boring because the interesting state is not in pods. The visualizer, Substrate Scope, renders the lifecycle directly: worker bays across the top, a restore queue below them, object storage at the bottom, and agent chips that physically move between the three as the cluster works. Below that, four telemetry charts compare reserved capacity against a pod-per-agent baseline measured from the real always-on agents in the same cluster.

It has two modes. Simulated mode runs in any browser with no cluster, which is useful for talks. Live mode polls the kagent controller and renders your actual cluster, and its buttons are real: workers +/- runs kubectl scale workerpool, SURGE fires one genuine chat at every agent, and AUTOSCALE turns on a demand-based scaler that resizes the real WorkerPool.

Standing it up on kind

Prerequisites: kind, kubectl, helm, Docker, and either Ollama running locally or an LLM API key. Every command below ran successfully on my machine (Apple Silicon, Docker Desktop).

Install Agent Substrate:

kind create cluster --name kagent-substrate

helm upgrade --install substrate-crds \
  oci://ghcr.io/kagent-dev/substrate/helm/substrate-crds \
  --version 0.0.6 --namespace ate-system --create-namespace --wait

helm upgrade --install substrate \
  oci://ghcr.io/kagent-dev/substrate/helm/substrate \
  --version 0.0.6 --namespace ate-system --wait --timeout 10m

Install kagent with the substrate integration. Two flags here are load-bearing and neither is in the official walkthrough: registry=ghcr.io (see the first gotcha below) and the Ollama provider, which needs no API key because the chart’s default host, host.docker.internal:11434, is exactly where kind on Docker Desktop finds your local Ollama:

helm upgrade --install kagent-crds \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \
  --version 0.9.9 --namespace kagent --create-namespace --wait

helm upgrade --install kagent \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --version 0.9.9 --namespace kagent --timeout 10m --wait \
  --set registry=ghcr.io \
  --set providers.default=ollama \
  --set providers.ollama.model=qwen3:4b \
  --set controller.substrate.enabled=true \
  --set controller.substrate.ateApiEndpoint=dns:///api.ate-system.svc:443 \
  --set controller.substrate.ateApiInsecure=true \
  --set substrateWorkerPool.create=true \
  --set substrateWorkerPool.replicas=2 \
  --set substrateWorkerPool.ateomImage=ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.6

Deploy an agent onto substrate. A SandboxAgent carries the same spec as a regular kagent Agent, plus a substrate section. Note platform: substrate, which the docs example omits and the API requires:

apiVersion: kagent.dev/v1alpha2
kind: SandboxAgent
metadata:
  name: sre-oncall
  namespace: kagent
spec:
  type: Declarative
  platform: substrate
  description: An SRE on-call assistant running as a substrate actor
  declarative:
    runtime: go
    modelConfig: default-model-config
    systemMessage: |
      You are an SRE on-call assistant. You triage alerts calmly.
      Keep answers to one or two sentences.
  substrate:
    workerPoolRef:
      name: kagent-default

The first reconcile bakes a golden snapshot, which takes about a minute, and then the agent is a stored snapshot that costs nothing until someone talks to it. The repo has a nine-agent fleet manifest, and the visualizer with a load generator:

node viz/server.mjs --live          # visualizer at http://localhost:8123
node viz/stimulate.mjs --budget 300 # real chats until the budget runs out

The --budget flag exists because I left the load generator running against the Anthropic API overnight and found out the hard way that it did not stop when I did.

Where the per-actor state lives

Kubernetes only sees WorkerPool and ActorTemplate. The state that makes the visualization interesting, which actor is running on which worker right now, lives in ateapi and reaches the outside world through a kagent controller endpoint:

kubectl port-forward -n kagent svc/kagent-controller 8083:8083
curl -s http://127.0.0.1:8083/api/substrate/status

The response carries four lists: workerPools, actorTemplates, actors (with live status like Resuming or Suspended), and workers (one row per worker pod, naming the actor it currently hosts). This is the same data the kagent UI’s Substrate page shows.

One lesson that cost me an evening: derive assignments from the workers list, not the actors list. Every completed session leaves a Suspended actor entry behind, so the actors list grows without bound (mine passed 900 entries in a day) and starts returning partial results under load. The workers list is never larger than your pool and it names the assigned agent directly. When my visualizer joined through the actors list, running sessions randomly disappeared from the board. When I switched to the workers list, they stopped disappearing.

What signal should autoscale a WorkerPool

CPU is the wrong signal for this runtime, and it is worth understanding why before someone wires up a standard HPA. Workers are slot-bound: one actor per worker regardless of load, and an LLM turn is mostly I/O wait on the model provider. I measured workers hosting active sessions at single-digit mCPU. A CPU-based autoscaler would sleep through total saturation and scale down under peak load.

What actually works is demand versus capacity:

  • Scale up on queue depth. Substrate rejects new sessions when the pool is full rather than queuing them, so sustained rejections are the purest statement of unmet demand. Today that signal only exists client-side, in your retry loop.
  • Scale down on the peak demand over a trailing window, not instantaneous demand, so a brief lull between sessions cannot slash the pool.
  • Go straight to the target in one jump. busy + queued is the number of workers you need. Stepping by one with long cooldowns just makes users wait through several cycles.

The autoscaler in the repo implements exactly that against the real CR, using the documented scaling path:

kubectl scale workerpools.ate.dev kagent-default -n kagent --replicas=4

One side effect to know about: kubectl scale takes field ownership of .spec.replicas, so later helm upgrade runs on the kagent chart will fail with a server-side apply conflict until you add --force-conflicts.

Gotchas

Everything in this section happened to me on kagent v0.9.9 and substrate v0.0.6, with the verbatim errors you would search for.

Agents stuck in Resuming forever

My first SandboxAgent sat in Resuming for 25 minutes. The generated ActorTemplate pins the declarative runtime image by digest, cr.kagent.dev/kagent-dev/kagent/golang-adk@sha256:e014..., and that digest no longer exists in that registry:

no such manifest: cr.kagent.dev/kagent-dev/kagent/golang-adk@sha256:e01479...

Substrate retries the pull forever and the agent never becomes Ready. The identical digest still exists on ghcr.io, and every kagent image is mirrored there, so the fix is one flag at install time: --set registry=ghcr.io. After that change, all nine golden snapshots baked in about a minute each.

spec.substrate may only be set when spec.platform is substrate

The docs example for SandboxAgent fails validation on 0.9.9:

The SandboxAgent "hello-substrate" is invalid: spec: Invalid value:
spec.substrate may only be set when spec.platform is substrate

Add platform: substrate to the spec. The field defaults to agent-sandbox.

SandboxAgents are not on the regular A2A endpoint

POST /api/a2a/kagent/<agent> returns Agent kagent/<agent> not found for SandboxAgents even when they are Ready. They live on a separate mount, and message/send requires a contextId:

curl -s -X POST http://127.0.0.1:8083/api/a2a-sandboxes/kagent/sre-oncall/ \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":"1","method":"message/send","params":{"message":{
        "kind":"message","messageId":"m1","contextId":"session-1","role":"user",
        "parts":[{"kind":"text","text":"Reply with the single word: ack"}]}}}'

Without contextId you get message contextId (session id) is required for substrate sandbox agents.

Worker pool has no free workers, but the pool looks idle

A session that gets killed mid-flight (a timeout, a cancelled chat, an API outage) can leave its actor pinning a worker slot indefinitely, sometimes as a ghost that no longer appears in the actor inventory at all. New sessions then fail with substrate worker pool has no free workers while the pool looks empty. The fix is cheap because snapshots live in object storage, not in the pods:

kubectl rollout restart deploy/kagent-default-deployment -n kagent

Only the wedged sessions die. Every agent’s snapshot survives.

In-sandbox egress fails until you restart substrate’s DNS

Actor egress resolves through substrate’s own CoreDNS deployment, and mine wedged with plugin/reload: Corefile changed but reload failed. Agents could not reach Ollama on the host until I restarted it:

kubectl rollout restart deploy/dns -n ate-system

Changing the model requires re-baking the fleet

Golden snapshots freeze the resolved model config at bake time. I switched the ModelConfig from Ollama to claude-haiku-4-5 and an existing agent kept answering from qwen. Delete and re-apply the SandboxAgents after any provider or model change; each re-bake takes about a minute and they queue through the pool.

The model switch was worth it for demos, for what it’s worth. Sessions went from 30 to 120 seconds on a local 4B model to 1.5 to 2 seconds end to end, restore included.

Wrapping up

Nine agents on two pods, real snapshot restores you can watch, and an autoscaler driven by the signal that actually reflects demand. The code, the fleet manifests, the visualizer, and the load generator are all in themsquared/kagent-substrate-demo. The deeper runtime internals are documented at learn.agentsubstrate.dev and the substrate repo.

Next on my list: wiring the queue-rejection signal into a proper KEDA scaler instead of my polling loop, and finding out whether substrate garbage-collects that ever-growing actor inventory. If you’re experimenting with agent runtimes on Kubernetes, this is a fun afternoon. I’ve been building agent tooling since the LLM agent playground, and this is the first runtime where the density story felt real on my own hardware.