Why Agent Substrate fails, in practice, is operability and policy, not checkpoint. I was wrong that a real process would make suspend a toy. I was right that kubectl, CPU autoscaling, a full pool with no scheduler, and ghost actors still shape how I run it.
I stood it up on a laptop with kagent : nine agents on two workers, then 18 on the same two workers, over a thousand real chat sessions, reserved capacity about 4.5x lower than always-on pods. Density worked. Failure modes did not vanish because density worked. The rest of this lives in the Agent Substrate field guide .
Why did I think Agent Substrate would fail?
The model asks you to believe five things at once. Checkpoint can serialize a real process. You can operate a fleet that is not in pods. Autoscaling still looks like Kubernetes. A small pool can be shared without a scheduler. Failure releases capacity.
I did not believe all five. I ran the runtime anyway, because idle reservation on always-on agents is a real line item: about 204 MiB and 50m CPU per default kagent agent, sitting still. Nine of those is roughly 1.8 GiB and 450m CPU reserved around the clock. If checkpoint was a toy, I wanted to know. If it was not, I wanted the list of what still breaks.
The one-line verdict, before the writeups: fail on resources the sandbox does not virtualize, fail on dashboards that only know pods, fail on CPU HPAs, fail on retry races, fail on ghost slots. Do not fail on “checkpoint is fake” for gVisor-managed state.
Does checkpoint fail on a real process?
I thought a held file lock, a socket, something ugly, would make suspend a toy. I tried flock(). It checkpointed. gVisor had already virtualized the lock into sentry state. The writeup
is me being wrong on purpose.
gVisor is an application kernel in userspace. The sentry reimplements file locking. An flock() taken inside the sandbox is sentry state, not an entry in the host kernel’s lock table, so it serializes like any other sentry-managed resource. I set out to test a resource that cannot be checkpointed, and I tested a resource gVisor had already made checkpointable. The experiment measured the sentry’s design, not Substrate’s.
I still believe the next class of failure: host-network sockets, bind-mounted devices, GPU memory, anything the sandbox does not own. GPUs for agents on Kubernetes is that bet applied to accelerators. Suspend-safe networking is that bet applied to TCP. Suspend cuts sockets in the versions I have run. If the agent is waiting on a long call, parking it is a failure, not a density win.
Verdict: fail on resources the sandbox does not virtualize, not on “checkpoint is fake.” Agent checkpoint to object storage is the storage contract. The runtime decides what to snapshot and when it is committed. The bucket moves bytes and should own integrity.
Can you operate Agent Substrate with kubectl?
kubectl get pods shows two quiet workers. I was right. I built a visualizer rather than pretend the CLI would grow an actors column. kubectl is lying to you
.
The interesting state is not a Pod. It is which actor is in which bay, restore queue depth, snapshot count, and rejections at the caller. Live assignment lives in ateapi. I read it through kagent’s /api/substrate/status and I treat the workers list as source of truth. The actors list grows without bound. Mine passed 900 entries in a day in the first lab and started returning partial results. Running sessions dropped off a board that joined through actors. The actors list lies. Trust the workers list.
If your on-call is pod-native and you do not add a workers view, Substrate will fail operationally even when it is working. That is a people failure I will keep attributing to the runtime, because the runtime chose to hide the workload. Substrate Scope is the board I built so I would stop explaining this with a whiteboard.
Why does CPU autoscaling fail on Substrate?
I was right. Slot-bound workers idle at single-digit mCPU. An LLM turn is mostly I/O wait on the model provider. Memory is not a better signal: restored snapshots have nearly constant footprints. A CPU HPA sleeps through saturation. It might scale down under peak load, because the graph looks quiet while every slot is busy.
Substrate rejects when the pool is full instead of queueing. The caller gets substrate worker pool has no free workers. Demand lives in retry loops, outside the runtime, where a standard HPA cannot see it. CPU is the wrong signal
.
On a serial model backend, adding workers made latency worse. More restored actors piled onto one generation queue. Everything timed out. The autoscaler made it worse . This is not a reason to skip Substrate. It is a reason to skip the first HPA example in every platform handbook, and to treat model concurrency as an upper bound on useful worker count.
Why is density without a scheduler a noisy-neighbor machine?
I was right, and I still think this is the structural hole. Full pool, reject, client retry, whoever hammers first wins. Scheduling agent sandboxes and who runs next when the agent pool is full .
Pods have pending queues and PriorityClass. Actors have a race. Kubernetes priorityClassName on the WorkerPool template decides whether a worker pod gets a node. It cannot pick which actor gets a slot. Every worker is identical at that layer. Every actor is invisible.
If you put two business units on one WorkerPool with no policy layer, Substrate did not fail. You used it as a scheduler. A batch agent with a tight retry loop will beat an incident responder with a polite backoff. I would rather add a resume queue I own, with a reserved floor for interactive work and fair share for the rest, than pretend autoscaling is a complete answer. Autoscaling delays the race. It does not decide winners at a replica ceiling, a GPU limit, or a model rate limit. Agent runtimes have an autoscaler, not a scheduler .
What failure modes pin capacity you think is free?
Ghost actors. Killed session, slot held, pool looks idle, every new session rejected. I hit this. A timeout, a cancelled chat, an API outage: the actor never released the worker. Sometimes it stopped showing up in the actor inventory and kept holding the slot anyway. On a two-worker pool, one ghost is half your capacity. Two is all of it.
Reset was cheap because snapshots live in object storage, not in the pods. Restart the WorkerPool deployment. Only the wedged sessions die. Every agent’s snapshot survives. Cheap is not the same as obvious. Ghost actors pin your workers . Scope has a RESET POOL button for the same reason.
The check I would automate: a worker that has been assigned for longer than your longest plausible session, with no turn activity, gets flagged. Then decide if you trust that enough to reset automatically. A board that reads the workers list shows a busy bay with nothing alive in it. A board that reads pods, or the actors list, shows a quiet cluster and a pile of rejections.
A short list of what I do with the five bets:
- I still pick Substrate when idle reservation is the cost I cannot pay. See what is Agent Substrate for the model.
- I pick a pod or agent-sandbox when the fleet is small or the process holds a GPU.
- I do not call it production until the production checklist is boring: real bucket, durable control plane, secrets at restore, sandbox class on purpose, demand-based scaling, reset runbook.
- I instrument occupancy from the workers list, rejections at the caller, restore time, and a reset path, before I put this in front of a pager.
The runtime is alpha in the versions I have used. APIs will move. That is not on the original five bets. It is the reason I keep the lab notes public rather than wrapping a platform SKU around them.
Field guide: Agent Substrate .