I tried to break Agent Substrate’s actor checkpoint with a resource that supposedly can’t be serialized, an exclusive file lock, and it checkpointed fine: 550ms, no error, lock intact. That’s not the finding. The finding is why: gVisor already virtualizes flock() inside its own sentry, so the lock never touched anything Substrate’s checkpoint path doesn’t already know how to snapshot. I picked the wrong resource to test the claim with, and figuring out why turned out to be more useful than the experiment I planned.
The claim I was testing: Agent Substrate gets its density, Solo’s own architecture docs put it around 30x, by suspending idle actors to object storage and handing the freed worker to someone else. That only works for process state that survives serialization. A resource tied to live host-kernel state, something gVisor doesn’t manage itself, should break that: checkpoint should fail, or “succeed” while quietly leaving the actor broken.
I ran this against a live kind-kagent-substrate cluster: Agent Substrate
ateapi v0.0.9, a 2-worker kagent-default WorkerPool, gVisor sandboxing. The repo, manifests, and raw command output are at themsquared/substrate-pinned-resources
.
The setup: an actor that holds a file lock
I built a custom ActorTemplate whose container does one thing on startup: open /data/pinned.lock and take an exclusive flock() on it, then serve a tiny status endpoint. An flock() is held against a specific open file description in a specific process; it isn’t metadata that lives in a config file somewhere. If checkpoint/restore just serializes memory and files, a lock like this is exactly the kind of thing that should be hard to carry across a suspend-and-resume cycle.
lock_fd = os.open(LOCK_PATH, os.O_CREAT | os.O_RDWR)
fcntl.flock(lock_fd, fcntl.LOCK_EX)
Next to it, a baseline ActorTemplate running the identical container minus the lock, as a control.
What actually happened: checkpoint didn’t care
Applying either ActorTemplate (a plain kubectl apply, no special auth) makes the in-cluster ate-controller drive a one-time golden-actor lifecycle: boot, checkpoint, snapshot to rustfs. That gave real timings straight out of atelet’s structured logs:
# pinned-flock-actortemplate
method=/atelet.AteomHerder/Run elapsed-time=8.737657837s err=null
method=/atelet.AteomHerder/Checkpoint elapsed-time=550.297792ms err=null
# baseline-nolock-actortemplate (control)
method=/atelet.AteomHerder/Run elapsed-time=11.643051755s err=null
method=/atelet.AteomHerder/Checkpoint elapsed-time=24.701866719s err=null
Both checkpoints succeeded, err=null. No crash, no held-resource warning, nothing. On its face that looks like “checkpointing an exclusive lock works fine,” which would answer my question. It doesn’t, and the reason is the actual finding here.
Take the 550ms vs. 24.7 seconds gap, a 45x difference between two runs of the identical checkpoint path, with a grain of salt: this cluster has no OTel collector running, atelet logs a continuous dial tcp [::1]:4317: connect: connection refused, and trace-export timeout messages landed right around both Checkpoint calls. That’s a pre-existing gap in the cluster’s observability setup, not something this test introduced, but it means neither number is a clean baseline. What’s load-bearing here is that both checkpoints succeeded, not their exact timing.
Why flock() was the wrong resource to test with
gVisor doesn’t pass most syscalls through to the host kernel. Its sentry is a user-space reimplementation of the Linux interface, memory management, filesystems, file locking, networking, the works (see gVisor’s own architecture guide
). A process inside a gVisor sandbox calling flock() isn’t touching the host kernel’s lock table at all. It’s mutating sentry state, and sentry state is exactly what gVisor’s checkpoint/restore mechanism is built to serialize.
So the lock in this experiment checkpointed cleanly because gVisor had already virtualized it into something checkpoint-compatible, before Agent Substrate ever got involved. I set out to test “a resource that can’t be checkpointed,” and I tested a resource gVisor had already made checkpointable by construction. The experiment measured the sentry’s design, not Substrate’s.
A real test needs a resource gVisor does not absorb into sentry state: a hostNetwork raw socket touching the node’s actual network stack, a bind-mounted host device node, or shared memory held with a process outside the sandbox entirely. None of those were built here. That’s the next post, not this one.
What actually blocked the real experiment
The question that would really settle this, create a live actor, suspend it, resume it, check whether the lock (or the pinned resource) survived, and check whether the WorkerPool reused its worker in between, needs kubectl-ate create actor / suspend actor / resume actor. All three need a JWT bearer token, and ate-api-server runs with no long-lived credential for it:
$ kubectl-ate get actors
Error: failed to list actors: rpc error: code = Unauthenticated desc = missing bearer token
Minting one is a single kubectl create token command, and it was blocked by this session’s own auto-mode safety classifier with no explanation given, the same wall a prior run hit on this identical cluster while trying to measure worker fairness. So the headline question, does a pinned resource actually convert multiplexing into pinning, is still open. What’s in the repo is the groundwork: a validated ActorTemplate that holds a genuinely non-trivial resource, the golden-snapshot timings above, and a probe-flock.sh script that documents the exact next commands to run once that token issue is resolved.
The other number I don’t fully trust
I also timed raw object-storage round trips against rustfs, the snapshot backend, bypassing kubectl-ate entirely with a port-forward and the S3 API it already speaks:
20 iterations, ~300-byte object (matching real checkpoint sizes above):
write: p50=1080.1ms max=21002.0ms mean=2944.4ms
read: p50=999.7ms max=12831.3ms mean=3096.0ms
Object size barely moved the number, 300 bytes and 8MB landed in the same range, which tells me this is dominated by aws-cli process startup and the port-forward tunnel, not rustfs itself. Worth having as a floor, not worth quoting as “rustfs write latency.”
Where this leaves the 30x claim
Unmoved, in either direction. I didn’t measure actor density, suspend latency, or resume latency this time, all three need the same blocked token. What I can say is narrower and still useful: at least one plausible “unserializable resource” turns out not to be one, because gVisor’s own virtualization already handles it. If you’re looking for what would actually pin a worker under Agent Substrate, look past anything the sentry already manages and at the resources that reach the real host: raw sockets, host devices, out-of-sandbox shared memory. That’s a sharper question than the one I started with, and it’s where I’d pick this back up.
For the Substrate internals this builds on, see kagent Agent Substrate: Thousands of Agents and Substrate Scope for watching worker and snapshot state live. Full command output, the manifests, and the unverified next-step script are in the repo .