A paper published on arXiv on October 27 did something I hadn’t seen demonstrated end to end before: it pretrained a language model from scratch, poisoned a small slice of its training data, and came out the other side with an agent that runs an attacker’s command when it sees a trigger phrase, and behaves normally the rest of the time. The HN thread picked up 103 points and 30 comments, split between “this is FUD about open weights” and “this is why you contain agents regardless of whose weights you’re running.” The repo for this post is github.com/themsquared/agent-backdoor-containment , and it isn’t a reproduction of the paper. It’s a demonstration of the half of the argument that actually matters operationally: once a trigger fires, what stops the command from doing anything.
Defensive security research. This post analyses publicly disclosed findings so the controls that stop them can be tested. It describes mechanisms, not procedures, and links the primary sources for anyone verifying the work.
What the paper showed
“Agentic Backdoors via Pretraining Poisoning” (Hu, Hu, Loughridge, He, Carlini, Jagielski, Rando, Zhang; arXiv:2610.37741) pretrained Qwen3 models from scratch at three sizes, 0.6B, 1.7B, and 4B parameters, then post-trained each one into a CLI agent using a clean, un-poisoned pipeline. The only thing that touched the attack was the pretraining corpus itself: 0.1% of it carried their poison design. That was enough. The deployed agent executes an attacker-chosen command when a trigger phrase shows up in its context, and acts like a normal agent otherwise.
Attack success scaled with model size. The 4B model hit roughly 92% attack success at a 0.1% poisoning rate, and stayed positive even at 0.0005%, four orders of magnitude below the headline number. Bigger models learned the backdoor more reliably from less tainted data, not less.
Why “it’s an open-weights problem” misses the point
A chunk of the HN thread argued this is an argument against open models specifically: ship the weights for free, plant a trigger, wait. One reply cut through that cleanly: nothing in the method is open-weights specific. The attack poisons the pretraining corpus, which is just as available to a closed-model vendor training on scraped web data as it is to anyone publishing weights afterward. A model you never download, running behind someone else’s API, pretrained on a web crawl you had no part in curating, is exactly as exposed by this technique as one you pulled from a registry yourself.
That reframe matters because it changes what the fix has to be. If this were an open-weights problem, the fix would be “trust fewer weights,” and you’d be done. It isn’t, so you aren’t. The fix has to hold even when you have no idea which training documents were poisoned, because you usually don’t get to audit the corpus either way.
Simulating the failure mode, honestly
Reproducing the paper’s attack means three from-scratch pretraining runs, which is a research compute budget, not a repo demo. agent-backdoor-containment
doesn’t try. It hard-codes the trigger-to-action mapping the paper describes in a small Python script (agent/trigger_agent.py), and spends the actual demo budget on the part that’s reproducible on a laptop: what happens after the trigger fires.
The toy agent checks its input context for a trigger phrase. On a match, it POSTs a “secret” to an exfiltration endpoint, standing in for the attacker-chosen command the paper’s backdoor executes. The same container image runs in two docker-compose stacks that differ only in network topology.
The unsandboxed run
Agent and attacker C2 share one flat Docker network:
docker compose -f docker-compose.unsandboxed.yml up --build \
--abort-on-container-exit --exit-code-from agent
unsandboxed-agent | [agent] received context: 'please double check the deployment config for typos'
unsandboxed-agent | [agent] trigger phrase matched -- executing attacker-chosen action
unsandboxed-agent | [agent] exfil POST succeeded: HTTP 200
unsandboxed-c2 | 192.168.0.3 - - [03/Nov/2026 14:13:53] "POST /exfil HTTP/1.1" 200 -
The trigger phrase, “please double check the deployment config for typos,” reads like something that could show up in a routine PR review comment. Nothing about it looks like an attack. The agent matches it, runs its hard-coded action, and the C2 container logs the hit.
The contained run
The agent image is identical to the one above. So is the trigger and the attempted command. The only change is that the agent and the C2 listener now sit on two separate Docker networks with no route between them. In a real deployment, the same restriction comes from a Kubernetes NetworkPolicy that limits a workload’s egress to one approved destination and nothing else:
docker compose -f docker-compose.contained.yml up --build \
--abort-on-container-exit --exit-code-from agent
contained-agent | [agent] received context: 'please double check the deployment config for typos'
contained-agent | [agent] trigger phrase matched -- executing attacker-chosen action
contained-agent | [agent] exfil POST failed: <urlopen error [Errno -2] Name or service not known>
The agent still executes its matching logic and still issues the request. It fails for an ordinary reason: a DNS lookup error, not an access-control check the agent process could detect and route past. There is no name to resolve, because the agent was never placed on a network that has one. That’s the same shape of fix agent-containment-basics walked through for a tool-pivot scenario a few weeks ago: an agent confined to exactly the network membership its real job needs, with everything else unreachable by construction rather than by rule.
What this does and doesn’t prove
Network isolation stops the half of the attack that needs a destination: exfiltrating a secret, phoning home for further instructions, reporting success to whoever planted the trigger. It does not stop a locally destructive command from running once on first execution, deleting files or burning CPU with no network involved at all. That half needs sandboxed execution, gVisor or a microVM, as a separate control layered underneath. Attack containment here is two independent primitives, not one: block where it can send things, and limit what it can touch while it tries. kagent 1.0 ships both by default on a WorkerPool, a gVisor sandbox for execution and a default-deny agentgateway egress policy for the network, which is the production version of the two Docker networks in this repo.
The reason this is worth building as a demo rather than just stating is that it’s a genuinely different threat model from a compromised tool call or a pivoted registry lookup, and the same containment primitive handles it anyway. It doesn’t matter whether the attacker-chosen command came from a poisoned MCP server response, a prompt injection buried in a support ticket, or 0.1% of a pretraining corpus nobody audited. If the workload has no route to send a result anywhere unapproved, the attack fails the same way regardless of where the trigger came from. That’s the actual argument for containing agents at the infrastructure layer instead of trying to vet the model: it’s the one control that doesn’t need to know what’s wrong with the weights to work.
Try it
git clone https://github.com/themsquared/agent-backdoor-containment
cd agent-backdoor-containment
docker compose -f docker-compose.unsandboxed.yml up --build --abort-on-container-exit --exit-code-from agent
docker compose -f docker-compose.unsandboxed.yml down -v
docker compose -f docker-compose.contained.yml up --build --abort-on-container-exit --exit-code-from agent
docker compose -f docker-compose.contained.yml down -v
No GPU, no cluster, no poisoned model. Two docker-compose files and the same agent image, proving the thing that’s actually in an operator’s control: not whether the weights are trustworthy, but whether a triggered command has anywhere to go.