# Thousands of AI Agents on Tens of Pods: kagent Agent Substrate

> kagent Agent Substrate is built to run thousands of AI agents on tens of pods. I stood it up on kind: snapshot restores, autoscaling, and every gotcha I hit.

- Canonical URL: https://webofmike.com/kagent-agent-substrate/
- Author: Mike Moore (https://webofmike.com/about/)
- Published: 2026-08-27
- Last modified: 2026-08-27
- Tags: Generative AI, Kubernetes, Tutorials, Platform Engineering
- Cite as: Mike Moore, "Thousands of AI Agents on Tens of Pods: kagent Agent Substrate", Web of Mike (webofmike.com), 2026-08-27. https://webofmike.com/kagent-agent-substrate/


Agent Substrate is a Kubernetes-native runtime that breaks the pod-per-agent model. Instead of one always-on pod per AI agent, idle agents are checkpointed to object storage as compressed snapshots and restored on demand into a small pool of pre-warmed [gVisor](https://gvisor.dev/) sandboxes. Agent count stops being a pod count: your fleet lives in object storage at near-zero marginal cost, and the design point is thousands of agents multiplexed across tens of pods. The kagent UI's own source comments size its substrate inventory page for "a cluster running four hundred thousand" actors, which tells you where the project thinks this goes.

I proved the mechanics end to end on a laptop: a kind cluster with [kagent](https://kagent.dev/), nine agents on two worker pods, over a thousand real chat sessions served, reserved capacity 4.5x lower than the same agents as always-on pods, and a live visualizer that shows every restore, queue wait, and checkpoint as it happens. The whole thing, visualizer included, is in [themsquared/kagent-substrate-demo](https://github.com/themsquared/kagent-substrate-demo) on GitHub. The ratio is the same machine at any scale; only the numbers get bigger.

## What Agent Substrate is

Always-on agent pods waste capacity because agents are idle most of the time. On my cluster, each of kagent's default always-on agents holds about 204 MiB of memory and requests 50m CPU while doing nothing. Nine of those is roughly 1.8 GiB resident and 450m CPU reserved around the clock.

[Agent Substrate](https://kagent.dev/docs/kagent/concepts/agent-substrate) decouples the agent's lifecycle from pod infrastructure:

1. When an agent is invoked, its actor is restored onto a free worker from a WorkerPool, rehydrated from a zstd snapshot in object storage.
2. The agent runs inside a gVisor sandbox for the duration of the session. One actor per sandbox, always.
3. When the session ends, the actor's state is checkpointed back to object storage and the worker slot frees for the next agent.

A worker hosts one actor at a time. The density comes from time-multiplexing: worker-0 in my cluster served 50+ sessions across nine different agents in an afternoon. Simultaneous sessions equal your worker count; total agents are limited only by object storage.

The runtime is two CRDs (`WorkerPool` and `ActorTemplate` in the `ate.dev` group) plus a control plane (`ateapi`, backed by a Valkey cluster), a data-plane router (`atenet`), a per-node snapshot mover (`atelet`), and a worker supervisor (`ateom`) that talks to gVisor's `runsc`. A detail I enjoyed: `atenet` is [agentgateway](https://agentgateway.dev/) under the hood. Its config lives at `/etc/agentgateway/config.yaml` inside the pod.

## What the visualizer shows

`kubectl get pods` makes substrate look boring because the interesting state is not in pods. The visualizer, Substrate Scope, renders the lifecycle directly: worker bays across the top, a restore queue below them, object storage at the bottom, and agent chips that physically move between the three as the cluster works. Below that, four telemetry charts compare reserved capacity against a pod-per-agent baseline measured from the real always-on agents in the same cluster.

It has two modes. Simulated mode runs in any browser with no cluster, which is useful for talks. Live mode polls the kagent controller and renders your actual cluster, and its buttons are real: workers +/- runs `kubectl scale workerpool`, SURGE fires one genuine chat at every agent, and AUTOSCALE turns on a demand-based scaler that resizes the real WorkerPool.

## Standing it up on kind

Prerequisites: `kind`, `kubectl`, `helm`, Docker, and either [Ollama](https://ollama.com/) running locally or an LLM API key. Every command below ran successfully on my machine (Apple Silicon, Docker Desktop).

Install Agent Substrate:

```bash
kind create cluster --name kagent-substrate

helm upgrade --install substrate-crds \
  oci://ghcr.io/kagent-dev/substrate/helm/substrate-crds \
  --version 0.0.6 --namespace ate-system --create-namespace --wait

helm upgrade --install substrate \
  oci://ghcr.io/kagent-dev/substrate/helm/substrate \
  --version 0.0.6 --namespace ate-system --wait --timeout 10m
```

Install kagent with the substrate integration. Two flags here are load-bearing and neither is in the official walkthrough: `registry=ghcr.io` (see the first gotcha below) and the Ollama provider, which needs no API key because the chart's default host, `host.docker.internal:11434`, is exactly where kind on Docker Desktop finds your local Ollama:

```bash
helm upgrade --install kagent-crds \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \
  --version 0.9.9 --namespace kagent --create-namespace --wait

helm upgrade --install kagent \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --version 0.9.9 --namespace kagent --timeout 10m --wait \
  --set registry=ghcr.io \
  --set providers.default=ollama \
  --set providers.ollama.model=qwen3:4b \
  --set controller.substrate.enabled=true \
  --set controller.substrate.ateApiEndpoint=dns:///api.ate-system.svc:443 \
  --set controller.substrate.ateApiInsecure=true \
  --set substrateWorkerPool.create=true \
  --set substrateWorkerPool.replicas=2 \
  --set substrateWorkerPool.ateomImage=ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.6
```

Deploy an agent onto substrate. A `SandboxAgent` carries the same spec as a regular kagent `Agent`, plus a substrate section. Note `platform: substrate`, which the docs example omits and the API requires:

```yaml
apiVersion: kagent.dev/v1alpha2
kind: SandboxAgent
metadata:
  name: sre-oncall
  namespace: kagent
spec:
  type: Declarative
  platform: substrate
  description: An SRE on-call assistant running as a substrate actor
  declarative:
    runtime: go
    modelConfig: default-model-config
    systemMessage: |
      You are an SRE on-call assistant. You triage alerts calmly.
      Keep answers to one or two sentences.
  substrate:
    workerPoolRef:
      name: kagent-default
```

The first reconcile bakes a golden snapshot, which takes about a minute, and then the agent is a stored snapshot that costs nothing until someone talks to it. The [repo](https://github.com/themsquared/kagent-substrate-demo) has a nine-agent fleet manifest, and the visualizer with a load generator:

```bash
node viz/server.mjs --live          # visualizer at http://localhost:8123
node viz/stimulate.mjs --budget 300 # real chats until the budget runs out
```

The `--budget` flag exists because I left the load generator running against the Anthropic API overnight and found out the hard way that it did not stop when I did.

## Where the per-actor state lives

Kubernetes only sees `WorkerPool` and `ActorTemplate`. The state that makes the visualization interesting, which actor is running on which worker right now, lives in `ateapi` and reaches the outside world through a kagent controller endpoint:

```bash
kubectl port-forward -n kagent svc/kagent-controller 8083:8083
curl -s http://127.0.0.1:8083/api/substrate/status
```

The response carries four lists: `workerPools`, `actorTemplates`, `actors` (with live status like `Resuming` or `Suspended`), and `workers` (one row per worker pod, naming the actor it currently hosts). This is the same data the kagent UI's Substrate page shows.

One lesson that cost me an evening: derive assignments from the `workers` list, not the `actors` list. Every completed session leaves a Suspended actor entry behind, so the actors list grows without bound (mine passed 900 entries in a day) and starts returning partial results under load. The workers list is never larger than your pool and it names the assigned agent directly. When my visualizer joined through the actors list, running sessions randomly disappeared from the board. When I switched to the workers list, they stopped disappearing.

## What signal should autoscale a WorkerPool

CPU is the wrong signal for this runtime, and it is worth understanding why before someone wires up a standard HPA. Workers are slot-bound: one actor per worker regardless of load, and an LLM turn is mostly I/O wait on the model provider. I measured workers hosting active sessions at single-digit mCPU. A CPU-based autoscaler would sleep through total saturation and scale down under peak load.

What actually works is demand versus capacity:

- Scale up on queue depth. Substrate rejects new sessions when the pool is full rather than queuing them, so sustained rejections are the purest statement of unmet demand. Today that signal only exists client-side, in your retry loop.
- Scale down on the peak demand over a trailing window, not instantaneous demand, so a brief lull between sessions cannot slash the pool.
- Go straight to the target in one jump. `busy + queued` is the number of workers you need. Stepping by one with long cooldowns just makes users wait through several cycles.

The autoscaler in the repo implements exactly that against the real CR, using the documented scaling path:

```bash
kubectl scale workerpools.ate.dev kagent-default -n kagent --replicas=4
```

One side effect to know about: `kubectl scale` takes field ownership of `.spec.replicas`, so later `helm upgrade` runs on the kagent chart will fail with a server-side apply conflict until you add `--force-conflicts`.

## Gotchas

Everything in this section happened to me on kagent v0.9.9 and substrate v0.0.6, with the verbatim errors you would search for.

### Agents stuck in Resuming forever

My first SandboxAgent sat in `Resuming` for 25 minutes. The generated ActorTemplate pins the declarative runtime image by digest, `cr.kagent.dev/kagent-dev/kagent/golang-adk@sha256:e014...`, and that digest no longer exists in that registry:

```text
no such manifest: cr.kagent.dev/kagent-dev/kagent/golang-adk@sha256:e01479...
```

Substrate retries the pull forever and the agent never becomes Ready. The identical digest still exists on ghcr.io, and every kagent image is mirrored there, so the fix is one flag at install time: `--set registry=ghcr.io`. After that change, all nine golden snapshots baked in about a minute each.

### spec.substrate may only be set when spec.platform is substrate

The [docs example](https://kagent.dev/docs/kagent/examples/agent-substrate) for SandboxAgent fails validation on 0.9.9:

```text
The SandboxAgent "hello-substrate" is invalid: spec: Invalid value:
spec.substrate may only be set when spec.platform is substrate
```

Add `platform: substrate` to the spec. The field defaults to `agent-sandbox`.

### SandboxAgents are not on the regular A2A endpoint

`POST /api/a2a/kagent/<agent>` returns `Agent kagent/<agent> not found` for SandboxAgents even when they are Ready. They live on a separate mount, and `message/send` requires a `contextId`:

```bash
curl -s -X POST http://127.0.0.1:8083/api/a2a-sandboxes/kagent/sre-oncall/ \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":"1","method":"message/send","params":{"message":{
        "kind":"message","messageId":"m1","contextId":"session-1","role":"user",
        "parts":[{"kind":"text","text":"Reply with the single word: ack"}]}}}'
```

Without `contextId` you get `message contextId (session id) is required for substrate sandbox agents`.

### Worker pool has no free workers, but the pool looks idle

A session that gets killed mid-flight (a timeout, a cancelled chat, an API outage) can leave its actor pinning a worker slot indefinitely, sometimes as a ghost that no longer appears in the actor inventory at all. New sessions then fail with `substrate worker pool has no free workers` while the pool looks empty. The fix is cheap because snapshots live in object storage, not in the pods:

```bash
kubectl rollout restart deploy/kagent-default-deployment -n kagent
```

Only the wedged sessions die. Every agent's snapshot survives.

### In-sandbox egress fails until you restart substrate's DNS

Actor egress resolves through substrate's own CoreDNS deployment, and mine wedged with `plugin/reload: Corefile changed but reload failed`. Agents could not reach Ollama on the host until I restarted it:

```bash
kubectl rollout restart deploy/dns -n ate-system
```

### Changing the model requires re-baking the fleet

Golden snapshots freeze the resolved model config at bake time. I switched the `ModelConfig` from Ollama to `claude-haiku-4-5` and an existing agent kept answering from qwen. Delete and re-apply the SandboxAgents after any provider or model change; each re-bake takes about a minute and they queue through the pool.

The model switch was worth it for demos, for what it's worth. Sessions went from 30 to 120 seconds on a local 4B model to 1.5 to 2 seconds end to end, restore included.

## Wrapping up

Nine agents on two pods, real snapshot restores you can watch, and an autoscaler driven by the signal that actually reflects demand. The code, the fleet manifests, the visualizer, and the load generator are all in [themsquared/kagent-substrate-demo](https://github.com/themsquared/kagent-substrate-demo). The deeper runtime internals are documented at [learn.agentsubstrate.dev](https://learn.agentsubstrate.dev/) and the [substrate repo](https://github.com/agent-substrate/substrate).

Next on my list: wiring the queue-rejection signal into a proper KEDA scaler instead of my polling loop, and finding out whether substrate garbage-collects that ever-growing actor inventory. If you're experimenting with agent runtimes on Kubernetes, this is a fun afternoon. I've been building agent tooling since [the LLM agent playground](/building-an-llm-agent-playground/), and this is the first runtime where the density story felt real on my own hardware.