# Agent platform at scale: the decisions that get expensive later > An agent platform at scale is not more pods. It is identity, scheduling, and suspend policy for fleets where most sessions are started by other agents. - Canonical URL: https://webofmike.com/agent-platform-at-scale/ - Author: Mike Moore (https://webofmike.com/about/) - Published: 2026-11-09 - Last modified: 2026-11-08 - Tags: Kubernetes, AI Agents, Platform Engineering - Cite as: Mike Moore, "Agent Platform at Scale", Web of Mike (webofmike.com), 2026-11-09. https://webofmike.com/agent-platform-at-scale/ An agent platform at scale is not more pods. It is identity, scheduling, and suspend policy for fleets where most sessions are started by other agents. Replica count is the KubeCon headline, not the part that gets expensive later. I have run 18 agents on two workers. The published lab was nine on two, over a thousand real chat sessions, reserved capacity about 4.5x lower than always-on pods. The kagent UI comments size an inventory page for hundreds of thousands of actors. I have not run that. I have run enough to see which decisions get expensive if you wait. ## What breaks first when agent count is not pod count? Density is the easy part. Always-on kagent defaults were about 204 MiB resident and 50m CPU request idle. Nine of those is roughly 1.8 GiB and 450m CPU reserved around the clock. Substrate time-multiplexes: one actor per worker, idle state in object storage. `kubectl get pods` goes quiet because the interesting state is not in pods. That is already an observability break. Watch the workers list, the restore path, and the snapshot shelf. [kubectl is lying to you](/kubectl-is-lying-to-you-watch-agent-substrate/). Scheduling is the next break. When the pool is full, Substrate rejects. Callers retry. The retry that lands first wins. Two teams sharing those two workers do not have a policy. They have a race. [Who runs next when the agent pool is full](/who-runs-next-when-the-agent-pool-is-full/). Autoscaling delays the race until a replica ceiling, a node pool, a GPU budget, or a model rate limit. Then something has to lose, and the runtime has no opinion. [Agent runtimes have an autoscaler, not a scheduler](/agent-runtimes-need-a-scheduler/). Ghost occupancy looks like scale and is a wedged slot. A killed session can pin a worker while the inventory looks empty. On a two-worker pool, one ghost is half your capacity. Reset has to be boring. [Ghost actors pin your workers](/ghost-actors-pin-your-workers/). The model backend is a separate capacity number. Extra workers on a serial generator made every turn slower. The queue moved from my callers to the model. I now treat generator concurrency as an upper bound on useful worker count. [The autoscaler made it worse](/autoscaler-made-it-worse-serial-model-backend/). ## Which trends should I take seriously before the fleet exists? **Power users.** One human with a pile of personal agents. If each agent is a pod, that human is a node-shaped bill. If each agent is a snapshot, they are a restore storm when they open the laptop. Idle policy and restore concurrency matter more than the headline agent count. Live sessions equal workers. Registered agents equal snapshots. [How many agents per Kubernetes pod](/how-many-agents-per-kubernetes-pod/). **Agent-to-agent fan-out.** Most sessions will not start from a person. They will start from another agent. That is a load generator you do not have an SLA for. Put A2A on a different pool, or a different priority class, before a recursive loop fills the interactive pool. SandboxAgents in the lab were not even on the regular A2A endpoint. They lived on a separate mount and `message/send` required a `contextId`. The protocol split is a hint: do not let unattended fan-out inherit the interactive SLO by accident. **Long-running case work.** Some sessions should stay suspended for days. That is a storage and identity problem: the snapshot must still exist, the credentials must not. Mint at resume. Drop at suspend. [Suspend and resume agents](/suspend-and-resume-agents/). Do not checkpoint long-lived secrets, GPU memory, or a TCP session the sandbox does not own. **Shared agents acting for someone.** Impersonation will show up as a feature request. Call it delegation and put the human in the token, or you will have no [proof of agent identity](/proof-of-agent-identity/) when it goes wrong. A WorkerPool ServiceAccount says which pod spoke. It does not say which actor. On a multiplexed runtime that lie gets worse with every restore. **Unattended automation.** No human in the loop means no "are you sure" and a much higher bar on [agent action verification](/agent-action-verification/). Do not ask a large model if the last step worked. Put a boolean on the call path. [Binary verification for agents](/binary-verification-for-agents/). ## What should I lock while Substrate is still cheap to change? Substrate is alpha in the versions I have used (kagent v0.9.9, substrate v0.0.6 in the published lab, v0.0.8 when I grepped the CRD). That is the right time to pick contracts. CRD names will move. The contracts should not. 1. **One WorkerPool per tenant, or one pool with a real scheduler.** Sharing without policy is how you get a platform-shaped incident. Separate pools are crude and work this week. A resume queue with an interactive floor and fair share on the remainder is the longer design. [Scheduling agent sandboxes](/scheduling-agent-sandboxes/). 2. **Snapshot bucket ownership.** Whose retention, whose encryption, whose restore drill. Suspend on node A, resume on node B, confirm it is the same session. A bucket outage should not be a control-plane outage you discover during an incident. 3. **A workers-list dashboard as a default, not a demo.** Occupancy, ghosts, and who holds a bay. Pod CPU will sleep through saturation. Workers hosting active sessions measured at single-digit mCPU. 4. **Model backend concurrency as a first-class capacity number, next to worker replicas.** Keep GPUs outside the actor. GPU memory does not snapshot. [GPUs for agents on Kubernetes](/gpus-for-agents-on-kubernetes/). 5. **Identity at restore.** Named agent, named session, human in the token if the work is delegated. Revoke the agent without revoking the person. 6. **Reject-vs-queue behavior.** The runtime rejects. If you need a queue, you own it. Write that down so the first internal customer does not discover it as a retry storm. 7. **Which pool interactive traffic is allowed to starve.** If you cannot name it, A2A will name it for you. I would not lock the CRD names. `WorkerPool` and `ActorTemplate` in `ate.dev` are what I have run. They will move. I would not lock a pretty UI. [Substrate Scope](/substrate-scope/) is nice. The workers list is sufficient. The enterprise bar is the same list with a sharper pass/fail: identity, isolation, scheduling, audit, reset, verification. Density is not on it. [Agent Substrate enterprise](/agent-substrate-enterprise/). ## What would I want the two-year retro to say? Not "we ran 20 agents per employee." That number is a thought experiment, not a measurement of mine. I want: we knew reservation was the cost, we did not scale on CPU, we did not let A2A share the on-call pool, and we could name the agent in the audit log. Size workers to simultaneous live sessions plus a buffer, not to registered agents. Size object storage to snapshots, including ones that stay suspended for a long time. Size the model backend separately. Compare reservation, not usage. Two quiet worker pods look like nothing. They held the live work. If every agent is busy, the ratio collapses to 1:1 and you should have used a [pod per agent](/pod-per-agent/). Substrate is the wrong complete answer for a handful of always-on processes that hold a GPU or a session socket. It is the right answer when idle time dominates and agent count tracks people, tickets, or other agents. ### Checklist: lock the cheap decisions before the first internal customer 1. Name who may share a WorkerPool. If the answer is "everyone," split pools now. 2. Write reject-vs-queue in the runbook. Count `no free workers` at callers. 3. Put A2A and interactive on different pools or different classes. 4. Own the snapshot bucket: retention, encryption, restore drill across nodes. 5. Mint identity at restore. Do not leave a long-lived secret in the snapshot. 6. Board the workers list by default. Treat a busy bay with no turn as a ghost. 7. Cap useful workers at model concurrency. Put verification on the gateway, not in the actor. If you are building that on Kubernetes, the [Agent Substrate field guide](/agent-substrate-field-guide/) is the rest of the how-to. The lab write-up is [kagent Agent Substrate](/kagent-agent-substrate/).