Running agents in production on Kubernetes takes a named identity for the agent, default-deny egress, a gate on tool calls, a demand signal that is not CPU, secrets that die with the session, a cost cap, an owner, and a way to freeze or kill without losing the log. The runtime you pick does not replace that list.
I like Agent Substrate. I still would not skip this list if I ran every agent as a Pod. The Substrate-shaped version is running Agent Substrate in production . This is the version I would tape next to either. If KubeCon week left you with a shopping list of runtimes, run this list against each of them. The one that makes the list shorter is the one I would trial. The one that only talks density is a lab.
Why isn’t a Deployment enough for production agents?
A Deployment keeps a process up. Kubernetes already knows how to schedule it, probe it, and restart it. That is necessary. It is not sufficient.
An agent is a process that will get a shell, call tools, spend tokens, and wait on a human. A ReplicaSet does not isolate tools, bound cost, or prove which agent acted. Those are platform controls around the process. Pod per agent is the default Kubernetes shape, and it is a fine prototype. It becomes a reservation bill when agent count tracks users or tickets.
I measured the reservation on kagent’s default always-on agents: about 204 MiB resident and 50m CPU sitting still. Nine of those is roughly 1.8 GiB and 450m CPU reserved around the clock. Substrate’s density (nine, then 18, agents on two workers, about 4.5x less reserved capacity) is one answer to that bill. It is not an answer to identity, egress, or audit. You can fail this list on Substrate and you can fail it on pods.
The fixture test still applies. Kill the sandbox mid-session. Does the session survive? Is every tool call still readable? If the answer is no, the record lives in the fixture. I wrote that from Ramp’s Laminar split and from my own clusters in the sandbox is the fixture, not the record . Production is the record living somewhere boring.
How do you name, isolate, and gate a production agent?
Name the agent. Not the Deployment. Not the user. The agent. Tokens scoped to it, logs tagged with it, a revoke path that does not disable the human. Proof of agent identity .
On a multiplexed runtime this is sharper: the worker’s ServiceAccount is the bay. If the audit log says system:serviceaccount:kagent:kagent-default, you have proof of a pool. On a pod-per-agent fleet the ServiceAccount is closer, and still not a per-session principal. Agent identity vs service account
is the fork. I want which agent, which session, on whose behalf, and what I can revoke.
Default-deny egress. The sandbox will get a shell. The shell will get curl. An allowlist that says “the internet” is not a control. Egress proxy , rate limits . Hostname allowlists fail on paste sites. DNS is a covert channel if you let it be.
NetworkPolicy still belongs on the pod as a backstop: workers cannot talk to the API unless you meant them to, egress from the pool goes only to the proxy. It cannot tell two actors on the same worker apart. Kubernetes network policy for agents . If you are testing on kind, remember kind’s default CNI accepts NetworkPolicy objects and then ignores them.
Gate tool calls. Network allow is not “delete ns prod.” Tool-call guardrails , agent action verification . A boolean on the call path: allowed or not, succeeded or not. Do not ask the same large model “did that work?” and treat “I am 90% sure” as a control. On a slot-bound runtime that second generation also holds a worker.
What demand signal and cost cap belong in production?
CPU is a lousy signal for a process that waits on a model. I measured Substrate workers hosting active sessions at single-digit mCPU. A CPU HPA sleeps through a full pool and may scale down under peak load. Count rejections, queue depth, or in-flight sessions. CPU is the wrong signal to autoscale agents .
This is not Substrate-only. A pod that spends the turn blocked on an API looks idle. If you scale that Deployment on CPU, you will shrink it while users wait. Measure demand where it is: sessions in flight, tokens in flight, rejections, queue depth.
Autoscaling is still not scheduling. At a replica ceiling, a GPU limit, or a provider rate limit, something has to lose. If you have no policy, the client that retries hardest wins. Agent runtimes have an autoscaler, not a scheduler . Scheduling agent sandboxes is the how-to I would put on top: a resume queue you own, a reserved floor for interactive work, fair share for the rest. Priority without quota is how everything becomes critical.
A cost cap. Tokens, not “we will watch Grafana.” If the loop can recurse, it will. Put the cap on the gateway. A dashboard is how you explain the bill. It is not how you stop the loop.
Secrets that outlive the turn are incidents. A long-lived provider key in a Secret is easy and wrong. Federation or short-lived exchange. Workload identity federation . On Substrate, mint at restore and drop at suspend so the snapshot does not contain the key. On a pod, the same rule: the process starts empty and asks for a credential scoped to this turn.
How do you freeze an agent without losing the record?
Kill without evidence is how you argue later. Snapshot or at least an append-only tool log. Stopping an agent without losing evidence .
I tested the arrangement on a kagent cluster with a gateway in front: session state in the control-plane database, per-request telemetry in a pipeline the gateway writes. Scale the Deployment to zero. Pods gone in seconds. The session, the prompt, the response, and the trace spans stayed queryable with no agent pods running. That only works when the agent process is not the system of record.
On Substrate the snapshot in object storage is a thicker freeze: park the actor, keep the object, read it later. Resetting a wedged WorkerPool is cheap for the same reason. Either way, the log the agent cannot edit is the one I would take to an incident review.
An owner. Every agent has a team that gets the page. “The platform” is not an owner. A page with no quota behind it trains everyone to mark their agent critical. Centrally define a small set of classes. Allocate them as a budget. The team still decides which of its agents gets one. They cannot decide that everything does.
How should you use this list when you pick a runtime?
Print it. Score each runtime by which rows it makes shorter, not by which logo was on a slide.
| Production row | Pod per agent | agent-sandbox | Agent Substrate |
|---|---|---|---|
| Process stays up | Deployment | sandboxed workload | WorkerPool slot |
| Identity | SA per pod, still not per session | same plus sandbox | must be per actor, SA is the pool |
| Egress | NetworkPolicy per pod | same | policy on the pod is the wrong grain |
| Demand signal | still not CPU | still not CPU | occupancy and rejections |
| Secrets | short-lived, not in the image | same | mint at restore, never in the snapshot |
| Freeze / kill | scale to zero, record outside | same | snapshot plus reset path |
| Density | you pay idle reservation | you still pay a sandbox per live agent | idle is a snapshot |
I have run the right-hand column on kind. I would not call that production until the Substrate checklist is done too. I would not skip this table if I stayed on the left.
A Monday-morning intake I would actually walk:
- Who is the agent, who owns the page, what can we revoke without disabling a human?
- Where does egress go, and what tool calls are denied before they run?
- What is the demand signal, the token cap, and the replica ceiling?
- Where does the record live if we kill it right now?
- What happens when two teams want the last slot?
If a vendor demo cannot answer those, you are buying a lab. Field guide: Agent Substrate . What is Agent Substrate if you need the runtime in one page first.