# Agent sandbox vs Agent Substrate: how I choose > Agent sandbox vs Agent Substrate: run agents as pods, isolate each one in a sandbox, or multiplex snapshot-backed actors onto a worker pool. How I choose. - Canonical URL: https://webofmike.com/agent-sandbox-vs-agent-substrate/ - Author: Mike Moore (https://webofmike.com/about/) - Published: 2026-11-09 - Last modified: 2026-11-08 - Tags: Kubernetes, AI Agents, Platform Engineering - Cite as: Mike Moore, "Agent Sandbox vs Agent Substrate", Web of Mike (webofmike.com), 2026-11-09. https://webofmike.com/agent-sandbox-vs-agent-substrate/ Agent sandbox vs Agent Substrate is a three-way choice: a plain pod, a per-agent sandbox, or multiplexed snapshot-backed actors on a worker pool. Isolation is a sandbox in both of the named options. Density, idle cost, and the control plane you have to operate are what change. I hit this as a validation error before I hit it as an architecture decision. On kagent 0.9.9 a `SandboxAgent` with a substrate section is invalid unless you set `spec.platform: substrate`. The field defaults to `agent-sandbox`. The API is telling you these are different runtimes that share a kind. If you only wanted the Substrate definition, start at [what is Agent Substrate](/what-is-agent-substrate/). This page is how I choose. ## What is the difference between a pod, agent-sandbox, and Agent Substrate? Three options, not two. **A pod.** One Deployment or Job per agent. Kubernetes already knows how to schedule, probe, and network it. You get `kubectl`, NetworkPolicy, and your existing HPA. You also get one reservation per agent, forever, including the 23 hours it is waiting on a human. That is the [pod per agent](/pod-per-agent/) problem. On my cluster, each of kagent's default always-on agents held about 204 MiB and requested 50m CPU while doing nothing. Nine of those is roughly 1.8 GiB resident and 450m CPU reserved around the clock. **agent-sandbox.** Keep the per-agent workload, put a sandbox around the process. Filesystem and syscall isolation get better. Density does not. You still pay a pod (or a sandbox VM) for each live agent. Use this when the threat model is "the agent has a shell" and the fleet is small. gVisor as an application kernel is a real isolation step: the sentry intercepts syscalls and the agent does not talk to the host kernel directly. That is still one reserved sandbox per agent. **Agent Substrate.** Agents become actors. Workers are a pool. Idle state is a zstd snapshot in object storage. Isolation is still a sandbox (gVisor in my lab). What changes is the unit of capacity: worker slots, not agent count. One actor per worker, always. Simultaneous sessions equal the worker count. Total agents are limited by object storage. I ran nine agents on two workers in the first lab, then 18 on the same two workers, across over a thousand real chat sessions. Reserved capacity was about 4.5x lower than the same agents as always-on pods. The shared kind is a trap. `SandboxAgent` looks like one API. `platform` is the fork. Default is agent-sandbox. Substrate is opt-in, and the API will reject a substrate section if you forget the field. I did. ## When should I run AI agents as Kubernetes pods? When count is in the single digits and will stay there. When the process holds a GPU or a session socket you cannot park. When I need a Service, a PDB, and a standard HPA, and I do not want a second control plane. A pod is the right default for three long-lived agents that run all day. Introducing snapshot storage, a WorkerPool, `ateapi`, and a restore path to save two reservations is a worse platform than the idle CPU those two pods waste. I am loyal to not reserving 50m CPU for an agent that is asleep. I am not loyal to multiplexed actors as a religion. Keep the pod when: - Agent count does not track users, tickets, or other agents. - Idle reservation is acceptable because the fleet is small. - The team already operates Deployments well, and on-call is pod-native. - The workload needs a stable IP, a probe, or a GPU inside the process. [GPUs for agents on Kubernetes](/gpus-for-agents-on-kubernetes/) is the GPU veto. [Suspend-safe networking](/suspend-safe-networking/) is the socket veto. That is also the agent-sandbox case if the threat model is untrusted code: isolate the pod, do not multiplex it. ## When is agent-sandbox the better choice than Agent Substrate? When you need a dedicated sandbox per session, you cannot tolerate restore latency, or you are not ready to operate snapshot storage and a worker pool. Substrate adds a control plane. Pay that cost when idle reservation is the thing you cannot afford. Restore is not free. The first reconcile bakes a golden snapshot, about a minute in my lab. A later restore is faster and still not a warm in-memory process. If your SLA is "first token in 100ms and the agent was cold," Substrate is the wrong tool until the snapshot is already warm on a worker. agent-sandbox keeps the process (or the sandbox VM) where you left it. You pay RAM for that privilege. agent-sandbox is also the better choice when you cannot share workers. Substrate time-multiplexes. The next actor lands on the same worker pod, in a fresh sandbox, after the previous one checkpoints out. Isolation is per-actor for the live turn. The worker's ServiceAccount, the node, and anything you mounted on the pool are still shared over time. If your threat model is "untrusted tenant code, dedicated kernel-level VM, never share the machine," look at a microVM sandbox class, not at skipping isolation, and maybe not at Substrate at all. Operate snapshot storage is a real job. Suspend and restore only work if object storage is someone else's problem you have already solved, or you are willing to solve it. My kind lab used an in-cluster store. Production is a real bucket, real integrity, real restore latency. [Suspend and resume agents](/suspend-and-resume-agents/) is the lifecycle. [Running Agent Substrate in production](/running-agent-substrate-in-production/) is the ops list. If that list is not boring yet, stay on agent-sandbox. ## How do I choose between them? I ask four questions, in this order. 1. **How many agents, and how often are they idle?** If the answer is "dozens, mostly waiting," Substrate's density is the point. If the answer is "three agents that run all day," a pod is simpler and I would not introduce a snapshot control plane to save two reservations. 2. **What is the isolation requirement?** Untrusted code, customer-supplied tools, or a shell: you need a sandbox either way. Substrate does not relax that. It multiplexes sandboxes. If you need a dedicated kernel-level VM per session and you cannot share workers, look at a microVM sandbox class, not at skipping isolation. 3. **Can you operate snapshot storage?** Suspend and restore only work if object storage is someone else's problem you have already solved, or you are willing to solve it. Production is a real bucket, real integrity, real restore latency. A kind cluster with an in-cluster store is a lab. 4. **Do you need scheduling, or just packing?** Substrate packs. It rejects when the pool is full. If two teams will fight over slots, you need a policy layer on top. That is [scheduling agent sandboxes](/scheduling-agent-sandboxes/), not a reason to go back to pods. Pods have the same fight, they just hide it in pending Pods. A practical default I actually use: - Prototype, one agent, one developer: a pod. - Untrusted tools, one session at a time, no density pressure: agent-sandbox. - Many agents, mostly idle, you can run object storage: Agent Substrate. - Untrusted plus many plus idle: Substrate with a strict sandbox class, and [network policy for agents](/kubernetes-network-policy-for-agents/) that does not assume one pod per agent. ## What tradeoffs should I expect after I pick Substrate? Restore latency is the first one. Warm workers exist so restore is not a cold container start. They still cost a pod. Size the pool to simultaneous sessions, not to total agents. [How many agents per Kubernetes pod](/how-many-agents-per-kubernetes-pod/) is the formula: live sessions equal workers, registered agents equal snapshots. The second tradeoff is observability. `kubectl get pods` shows the pool. It does not show the agents. If your on-call culture is pods and Prometheus red/green, you will think the cluster is idle while actors are restoring. Read [kubectl is lying to you](/kubectl-is-lying-to-you-watch-agent-substrate/) before you put this in front of a pager. I built [Substrate Scope](/substrate-scope/) because I got tired of explaining that with a whiteboard. The third is identity. A sandboxed actor with a broad token is still a broad token. On a multiplexed pool, the worker's ServiceAccount is the pool. [Proof of agent identity](/proof-of-agent-identity/) has to name the agent, not the bay it happened to land in. The fourth is scheduling. Full pool, reject, client retry, whoever hammers first wins. That is packing working as designed. It is a noisy-neighbor machine if two business units share one WorkerPool with no policy. [Agent runtimes have an autoscaler, not a scheduler](/agent-runtimes-need-a-scheduler/) is the argument. Do not treat that as a reason to go back to pods unless you also liked pending Pods as a scheduler. A checklist I run before I switch a fleet: - Count agents and idle fraction. If both are small, stop. Stay on a pod. - Name the isolation class: gVisor, microVM, or "the process has a shell and we accept that." - Confirm object storage is a bucket you already back up, not emptyDir. - Confirm restore latency is acceptable for a cold agent. Golden snapshot bake is about a minute in my lab. Do not promise a first-token SLO you have not measured. - Confirm you have a workers-list view, a rejection counter at the caller, and a reset runbook for ghost actors. - Confirm you have an identity and egress story that follows the actor, not the worker pod. [Sandboxed networking for agents](/sandboxed-networking-for-agents/). I am not loyal to the multiplexed shape. I am loyal to not reserving 50m CPU for an agent that is asleep. The rest of this lives in the [Agent Substrate field guide](/agent-substrate-field-guide/).