# Does Agent Substrate's Checkpoint Survive a Held File Lock?

> I tried to break Agent Substrate's checkpoint with a held file lock. It survived: gVisor virtualizes flock() inside its own sentry, not the host kernel.

- Canonical URL: https://webofmike.com/substrate-pinned-resources/
- Author: Mike Moore (https://webofmike.com/about/)
- Published: 2026-11-04
- Last modified: 2026-11-04
- Tags: Kubernetes, Platform Engineering, Generative AI, Code
- Cite as: Mike Moore, "Does Agent Substrate's Checkpoint Survive a Held File Lock?", Web of Mike (webofmike.com), 2026-11-04. https://webofmike.com/substrate-pinned-resources/


I tried to break Agent Substrate's actor checkpoint with a resource that supposedly can't be serialized, an exclusive file lock, and it checkpointed fine: 550ms, no error, lock intact. That's not the finding. The finding is why: gVisor already virtualizes `flock()` inside its own sentry, so the lock never touched anything Substrate's checkpoint path doesn't already know how to snapshot. I picked the wrong resource to test the claim with, and figuring out why turned out to be more useful than the experiment I planned.

The claim I was testing: Agent Substrate gets its density, Solo's own architecture docs put it around 30x, by suspending idle actors to object storage and handing the freed worker to someone else. That only works for process state that survives serialization. A resource tied to live host-kernel state, something gVisor doesn't manage itself, should break that: checkpoint should fail, or "succeed" while quietly leaving the actor broken.

I ran this against a live `kind-kagent-substrate` cluster: [Agent Substrate](https://github.com/kagent-dev/substrate) `ateapi` v0.0.9, a 2-worker `kagent-default` WorkerPool, gVisor sandboxing. The repo, manifests, and raw command output are at [themsquared/substrate-pinned-resources](https://github.com/themsquared/substrate-pinned-resources).

## The setup: an actor that holds a file lock

I built a custom `ActorTemplate` whose container does one thing on startup: open `/data/pinned.lock` and take an exclusive `flock()` on it, then serve a tiny status endpoint. An `flock()` is held against a specific open file description in a specific process; it isn't metadata that lives in a config file somewhere. If checkpoint/restore just serializes memory and files, a lock like this is exactly the kind of thing that should be hard to carry across a suspend-and-resume cycle.

```python
lock_fd = os.open(LOCK_PATH, os.O_CREAT | os.O_RDWR)
fcntl.flock(lock_fd, fcntl.LOCK_EX)
```

Next to it, a baseline `ActorTemplate` running the identical container minus the lock, as a control.

## What actually happened: checkpoint didn't care

Applying either `ActorTemplate` (a plain `kubectl apply`, no special auth) makes the in-cluster `ate-controller` drive a one-time golden-actor lifecycle: boot, checkpoint, snapshot to rustfs. That gave real timings straight out of `atelet`'s structured logs:

```
# pinned-flock-actortemplate
method=/atelet.AteomHerder/Run        elapsed-time=8.737657837s   err=null
method=/atelet.AteomHerder/Checkpoint elapsed-time=550.297792ms   err=null

# baseline-nolock-actortemplate (control)
method=/atelet.AteomHerder/Run        elapsed-time=11.643051755s  err=null
method=/atelet.AteomHerder/Checkpoint elapsed-time=24.701866719s  err=null
```

Both checkpoints succeeded, `err=null`. No crash, no held-resource warning, nothing. On its face that looks like "checkpointing an exclusive lock works fine," which would answer my question. It doesn't, and the reason is the actual finding here.

Take the 550ms vs. 24.7 seconds gap, a 45x difference between two runs of the identical checkpoint path, with a grain of salt: this cluster has no OTel collector running, `atelet` logs a continuous `dial tcp [::1]:4317: connect: connection refused`, and trace-export timeout messages landed right around both `Checkpoint` calls. That's a pre-existing gap in the cluster's observability setup, not something this test introduced, but it means neither number is a clean baseline. What's load-bearing here is that both checkpoints succeeded, not their exact timing.

## Why flock() was the wrong resource to test with

gVisor doesn't pass most syscalls through to the host kernel. Its sentry is a user-space reimplementation of the Linux interface, memory management, filesystems, file locking, networking, the works (see gVisor's own [architecture guide](https://gvisor.dev/docs/architecture_guide/intro/)). A process inside a gVisor sandbox calling `flock()` isn't touching the host kernel's lock table at all. It's mutating sentry state, and sentry state is exactly what gVisor's checkpoint/restore mechanism is built to serialize.

So the lock in this experiment checkpointed cleanly because gVisor had already virtualized it into something checkpoint-compatible, before Agent Substrate ever got involved. I set out to test "a resource that can't be checkpointed," and I tested a resource gVisor had already made checkpointable by construction. The experiment measured the sentry's design, not Substrate's.

A real test needs a resource gVisor does **not** absorb into sentry state: a `hostNetwork` raw socket touching the node's actual network stack, a bind-mounted host device node, or shared memory held with a process outside the sandbox entirely. None of those were built here. That's the next post, not this one.

## What actually blocked the real experiment

The question that would really settle this, create a live actor, suspend it, resume it, check whether the lock (or the pinned resource) survived, and check whether the WorkerPool reused its worker in between, needs `kubectl-ate create actor` / `suspend actor` / `resume actor`. All three need a JWT bearer token, and `ate-api-server` runs with no long-lived credential for it:

```
$ kubectl-ate get actors
Error: failed to list actors: rpc error: code = Unauthenticated desc = missing bearer token
```

Minting one is a single `kubectl create token` command, and it was blocked by this session's own auto-mode safety classifier with no explanation given, the same wall a prior run hit on this identical cluster while trying to measure worker fairness. So the headline question, does a pinned resource actually convert multiplexing into pinning, is still open. What's in the repo is the groundwork: a validated `ActorTemplate` that holds a genuinely non-trivial resource, the golden-snapshot timings above, and a `probe-flock.sh` script that documents the exact next commands to run once that token issue is resolved.

## The other number I don't fully trust

I also timed raw object-storage round trips against rustfs, the snapshot backend, bypassing `kubectl-ate` entirely with a port-forward and the S3 API it already speaks:

```
20 iterations, ~300-byte object (matching real checkpoint sizes above):
write: p50=1080.1ms  max=21002.0ms  mean=2944.4ms
read:  p50=999.7ms   max=12831.3ms  mean=3096.0ms
```

Object size barely moved the number, 300 bytes and 8MB landed in the same range, which tells me this is dominated by `aws-cli` process startup and the port-forward tunnel, not rustfs itself. Worth having as a floor, not worth quoting as "rustfs write latency."

## Where this leaves the 30x claim

Unmoved, in either direction. I didn't measure actor density, suspend latency, or resume latency this time, all three need the same blocked token. What I can say is narrower and still useful: at least one plausible "unserializable resource" turns out not to be one, because gVisor's own virtualization already handles it. If you're looking for what would actually pin a worker under Agent Substrate, look past anything the sentry already manages and at the resources that reach the real host: raw sockets, host devices, out-of-sandbox shared memory. That's a sharper question than the one I started with, and it's where I'd pick this back up.

For the Substrate internals this builds on, see [kagent Agent Substrate: Thousands of Agents](/kagent-agent-substrate/) and [Substrate Scope](/substrate-scope/) for watching worker and snapshot state live. Full command output, the manifests, and the unverified next-step script are in the [repo](https://github.com/themsquared/substrate-pinned-resources).