Scheduling sandboxes for RL is a multiplexing problem: each training step fans out into sandboxes that spend most of their wall time waiting on a model. Suspend the actor during that call and a small worker pool can cover a large rollout set. Give every rollout a pod and you have reserved a cluster for sleep.
This is the first place I would expect Agent Substrate’s density story to be obviously true, and the first place I would still demand a benchmark. I have not published a SWE-bench-on-Substrate number in this guide. I have multiplexed chat agents that were idle on the model for the same reason, and I have watched extra workers make a serial generator worse. That is enough to write the how-to without borrowing someone else’s 30% wall-time claim.
Why does multiplexing pay while the sandbox waits on a model?
The sandbox is idle. The wait is the model. Restore is cheaper than the wait.
That is the common case for tool-using agents sitting on a generation. It is also the RL case if the environment step is light (edit a file, run a small test) and the policy model is heavy. The actor holds a worker slot for the whole turn if you never suspend. On Substrate, a worker is a bay: one actor at a time. Idle time is the whole density story. I ran 18 chat agents on two workers, and the published lab was nine on two, reserved capacity about 4.5x lower than always-on pods. Always-on kagent defaults were about 204 MiB resident and 50m CPU request idle. Nine of those is roughly 1.8 GiB and 450m CPU reserved around the clock. RL rollouts that look like those chat agents will waste the same reservation if each one is a pod.
Workers hosting active sessions measured at single-digit mCPU. The process is waiting on tokens. A CPU graph will tell you the cluster is quiet while every slot is taken. CPU is the wrong signal to autoscale agents . Measure wall time of the generation step.
It does not pay when:
- the rollout is compiling or running a GPU kernel inside the sandbox
- checkpoint and restore is slower than the LLM call (tiny prompts, huge snapshots)
- you need suspend-safe networking and you do not have it, and the env holds a socket
I do not suspend mid-call in the versions I have run. Suspend cuts TCP. Drain the tool, then park. Multiplexing during model waits is this post. Multiplexing during tool waits needs suspend-safe networking to exist. If a third of your agents are blocked on tools at any time, those slots are busy even though the LLM is idle. Size the pool for in-flight calls, not only for thinking.
Keep GPUs on the model side. If the actor holds the device, you cannot snapshot GPU memory, and you have thrown away the density the runtime exists for. GPUs for agents on Kubernetes .
How should I run the comparison against one pod per rollout?
Same agent loop. Same model endpoint. Same number of rollouts. Do not change the prompt set halfway through and call it a runtime result.
Pool A: one pod (or one always-on sandbox) per rollout.
Pool B: Substrate workers much less than rollouts, suspend during the LLM call.
Collect:
- step wall time (p50, p95)
- reserved millicores and memory, not usage
- restore errors and ghost slots
- model queue depth
If B wins on reservation and loses on wall time, you saturated the generator. That is not a Substrate failure. Stop adding workers. I already had extra workers pile onto a serial generator and time out. The queue moved from my callers’ retry loops to the model, where the board could not see it. I capped load-generator concurrency to 2 against Ollama to match what that backend could serve. The autoscaler made it worse .
If B loses on both, restore is too expensive or suspend is dropping work. Fix that before you talk about multiplexing. Ghost occupancy looks like a full pool: substrate worker pool has no free workers while the inventory looks idle. Reset the workers. Snapshots survive. Ghost actors pin your workers
.
I am not going to turn 18:2 into a training-cluster ratio. You cannot take a chat-agent laptop number and multiply it across RL rollouts without measuring restore, the model queue, and ghost slots. My sessions were chat turns against a local or API model. RL rollouts and long tool calls have different idle shapes. Compare reservation, not usage. Two quiet worker pods look like nothing. They held the live work. The other agents were on the shelf. kubectl is lying to you .
What if the worker pool is full in the middle of a step?
A full pool plus retries means an arbitrary subset of rollouts make progress. In RL, that is a biased batch. You want a resume queue with a policy, not a race.
Scheduling agent sandboxes is the general design: a queue you own, a free-worker buffer, classes as a Kubernetes object. For training I would start with fair share across rollouts in the step, not “interactive beats batch,” because here the rollouts are the job. An incident-shaped retry race is who runs next when the agent pool is full . Do not put the training loop on the same WorkerPool as on-call chat and hope jitter will sort it.
The runtime will not help you here. Substrate rejects. The caller retries. The retry that lands first wins. That is packing working as designed. It is also a silent change to which trajectories enter the gradient.
A free-worker buffer still matters. I already scale with a floor of 2 on a laptop pool. Restore is bursty. A step that wakes every rollout at once is a restore storm. Never run at 100% occupancy if you care which rollouts land.
Do not scale the WorkerPool as if it were the only bottleneck. Model backend concurrency is an upper bound on useful worker count. If the generator is serial, more workers is a more expensive waiting room.
When should I refuse to multiplex a rollout?
When the sandbox is the compute, not the wait. Compile, test, in-process GPU: the actor is busy. A snapshot of a busy sandbox is not density. It is a pause button with a bill.
When you cannot park the connection. A database transaction, a multiplexed gRPC stream, a websocket: suspend is a reset. Wait it out, or run the I/O from a proxy that outlives the actor.
When the dedicated always-on shape is actually what you wanted. A handful of long-lived trainers that hold a device or a session socket should stay a pod per agent , or the agent-sandbox vs Agent Substrate fork. Substrate is the wrong tool if every rollout is live all the time. The ratio collapses to 1:1 and you paid for a snapshot control plane to get back to pods.
Checklist: decide if multiplexing this RL step is worth it
- Time one environment step with the model call included. If most of the wall time is the LLM, keep going. If most of it is compile, test, or GPU in-process, stop.
- Confirm you can suspend between the tool return and the next generation, not mid-socket.
- Run Pool A vs Pool B with the same agent, model, and step count.
- Record reserved CPU and memory, step p50 and p95, restore errors, ghost slots, and generator queue depth.
- If wall time got worse while reservation got better, cap workers at the model’s concurrency. Do not add bays in front of one cashier.
- Put a fair-share resume policy in front of the pool so a full step cannot drop an arbitrary subset of rollouts.
- Keep inference outside the actor. Do not snapshot a GPU you do not own.
Field guide: Agent Substrate . The lab numbers behind the density claim are in kagent Agent Substrate .