What is Agent Substrate?

What is Agent Substrate? It is a Kubernetes-native runtime that treats an AI agent as a snapshot-backed actor, not as a Deployment. Idle agents are checkpointed to object storage as zstd snapshots and restored onto a small pool of pre-warmed gVisor sandboxes when a request arrives, so agent count stops being a pod count.

That is the whole idea. Agents are idle most of the time. An always-on pod still reserves CPU and memory while it waits on a human, a tool, or a model. Substrate parks that idle state in object storage and keeps a small pool of workers warm. A request restores the actor, the turn runs inside a sandbox, and the actor goes back to storage.

I wrote the first lab as thousands of agents on tens of pods . This page is the shorter answer I wanted when I started. The rest of the series lives in the Agent Substrate field guide .

What problem does Agent Substrate solve?

Kubernetes wants a Pod. An agent is a process that needs isolation, a network path, and a restart story, so a Deployment looks like the honest default. Then someone asks for one agent per user, or one per ticket, and you have a ReplicaSet factory. Each replica keeps requesting CPU and memory for as long as it exists, including the hours it is waiting on a human.

On my cluster, each of kagent’s default always-on agents held about 204 MiB resident and requested 50m CPU while doing nothing. Nine of those is roughly 1.8 GiB resident and 450m CPU reserved around the clock. Those pods look cheap on a CPU graph. They are not cheap on the request. Idle pods barely use CPU and you still pay for the reservation.

Substrate’s answer is to stop treating agent count as pod count. The fleet lives in object storage at near-zero marginal cost. The live capacity is a WorkerPool. Simultaneous sessions equal the worker count. Total agents are limited by snapshots. That is the pod per agent problem, solved as packing, not as a smaller Deployment.

The kagent docs put the same split in their own words: decouple the agent’s lifecycle from pod infrastructure, snapshot idle actors, rehydrate them on demand onto pre-warmed workers. I measured the mechanics. I did not invent the model.

How does Agent Substrate run an agent without a dedicated pod?

Two CRDs in the ate.dev group: WorkerPool and ActorTemplate. The pool is the hardware. The template is the workload. Kubernetes only sees those objects plus a Deployment for the pool. The interesting live state is not a Pod status.

On a running cluster the control plane is ateapi (backed by Valkey in my lab). atenet routes into actors. atelet moves snapshots on the node. ateom supervises the worker and talks to the sandbox. I mapped that path in Agent Substrate dataplane . The kagent concept page names the same four processes, plus a reconciler for the CRDs. I watch the four, because those are the ones that break.

The sandbox I run is gVisor . One actor per sandbox, always. That is isolation, not packing. gVisor is an application kernel in userspace: the sentry reimplements the Linux syscall interface, so the agent talks to gVisor, not to the host kernel. Density still comes from time-multiplexing. Worker-0 in my first lab served 50+ sessions across nine different agents in an afternoon. Two agents do not share a sandbox at the same time.

The invoke path is short:

1. A request arrives for an agent.
2. If a worker is free, Substrate restores the actor from a zstd snapshot in object storage onto that worker.
3. The turn runs inside the sandbox.
4. When the session ends, the actor is checkpointed back to object storage and the slot frees for the next agent.

If no worker is free, the runtime rejects. The caller retries. That is not a queue. I wrote that gap in agent runtimes have an autoscaler, not a scheduler .

A SandboxAgent on kagent carries the same spec as a regular Agent, plus a substrate section. Two flags are load-bearing. Pin the image registry you can actually pull (registry=ghcr.io was that pin in my helm install). Set platform: substrate, or the API rejects the sandbox spec because the field defaults to agent-sandbox. I hit that as a validation error before I hit it as an architecture decision. Agent sandbox vs Agent Substrate is the choice.

What did I actually measure on a laptop cluster?

On a laptop kind cluster:

• nine agents on two workers in the first lab, then 18 on the same two workers for the later setup
• over a thousand real chat sessions
• reserved capacity about 4.5x lower than the same agents as always-on pods

Those numbers are mine. If someone else’s abstract claims a different efficiency ratio, treat it as their benchmark, not as a constant of the runtime. Compare reservation, not usage.

A worked example with the numbers I already published. Nine always-on kagent defaults: about 204 MiB each, 50m CPU request each. That is roughly 1.8 GiB resident and 450m CPU reserved around the clock, whether anyone is talking to them or not. The same nine agents as Substrate actors share two worker pods. Idle agents are snapshots. Live turns occupy a worker slot. When I later doubled the fleet to 18 agents, I did not double the worker count. The extra agents lived on the snapshot shelf.

I watch this with Substrate Scope , Apache-2.0, simulated and live modes. The board has worker bays across the top, a restore queue under them, and the snapshot shelf at the bottom. kubectl get pods shows two quiet workers. Scope shows the agents moving. That is why kubectl is lying to you .

The kagent UI’s own source comments size its substrate inventory page for “a cluster running four hundred thousand” actors. That is where the project thinks this goes. It is not a number I measured. Do not quote it as my result.

What is Agent Substrate not?

It is not a scheduler. When the pool is full, Substrate rejects. The caller retries. Whoever lands first wins the slot. Pods have pending queues and PriorityClass. Actors have a race. If two teams share one WorkerPool with no policy layer, you used a packer as a scheduler. The how-to sits in scheduling agent sandboxes .

It is not a pod-shaped object you can see with kubectl get pods. The interesting state is actors, workers, the restore queue, and the snapshot shelf. Live assignment (which actor is on which worker) lives in ateapi. I read it through the kagent controller:

kubectl port-forward -n kagent svc/kagent-controller 8083:8083
curl -s http://127.0.0.1:8083/api/substrate/status

Four lists: worker pools, actor templates, actors, workers. Use the workers list. The actors list grows without bound because every finished session leaves a Suspended entry behind. I learned that the hard way.

It is not a substitute for identity or egress policy. A sandboxed actor with a broad token is still a broad token. On a multiplexed pool, the worker’s ServiceAccount is the pool, not the agent. See proof of agent identity and sandboxed networking for agents .

It is not a GPU runtime. The snapshot is process memory and files the sandbox owns. GPU memory lives on the device. I keep inference outside the actor. GPUs for agents on Kubernetes is that veto.

It is not a pause-a-Pod primitive. Kubernetes has no “ship this process’s RAM to object storage and bring it back on another node” as a Pod feature. Substrate does that for an actor. The worker is a bay. The snapshot is the agent. Suspend and resume agents is the lifecycle.

How do I stand Agent Substrate up?

Kind, helm, a WorkerPool with two replicas, then one agent. The first reconcile bakes a golden snapshot, about a minute in my lab. After that the agent costs storage until someone talks to it. The commands I actually ran are in the lab post . The demo repo is themsquared/kagent-substrate-demo . The visualizer is themsquared/substrate-scope .

A checklist I would run before I believed the cluster:

• WorkerPool replicas equal the sessions that must be live at once, not the number of agents you defined. How many agents per Kubernetes pod is the sizing post.
• platform: substrate is on every SandboxAgent. Without it, kagent 0.9.9 rejects spec.substrate.
• Image registry is one you can pull. My first agent sat in Resuming for 25 minutes on a digest that only still existed on ghcr.io.
• Golden snapshot exists before you send traffic. First bake is slow. Later restore is a snapshot rehydrate, not a cold pod boot.
• Chat the agent on the sandbox A2A mount, not the regular /api/a2a/ path. SandboxAgents are not on the regular endpoint, and message/send wants a contextId.
• Occupancy comes from the workers list. Rejections come from the caller (substrate worker pool has no free workers). If you only have pod CPU, you have the wrong dashboard.
• Reset path exists. A killed session can leave a ghost actor pinning a worker while the pool looks idle. Restarting the WorkerPool was cheap because snapshots live in object storage.

If you are still choosing between a pod, agent-sandbox, and Substrate, start at agent-sandbox vs Agent Substrate . This page is the runtime once you have already picked it. If you want the ops list, not the definition, that is running Agent Substrate in production .