Running Factory Droid as a kagent Agent on Agent Substrate

I wanted to know whether a third-party coding agent could run as a first-class kagent agent, inside an Agent Substrate sandbox, doing platform engineering work. It can. Factory’s Droid now answers through kagent’s API, calls read-only Kubernetes tools over MCP, remembers the previous turn, and never sees its own API key. The code is at github.com/themsquared/kagent-droid.

It took two tries. The first version, built on Factory’s Python SDK, worked on my laptop and failed every session on the cluster. Why it failed is most of what this post is about.

Everything below ran on kind with kagent 1.0 (an Enterprise pre-release build, 1.0.0-alpha4), Agent Substrate 0.2.0-beta5, gVisor release-20260824.0 on arm64, and droid 0.242.0.

How a BYO agent works in kagent v1

kagent v1 runs agents on Substrate actors instead of Deployments. I covered that model when v1.0.0-alpha1 shipped with its Claude Code harness. Alongside the native Claude Code and Codex harnesses there is a byo runtime: give kagent an image, and it treats the inside as opaque as long as the image speaks its contract.

The contract is small:

• A2A over gRPC, service lf.a2a.v1.A2AService, on port 80. A JSON-RPC-only image is never reached.
• GET /readyz on port 8081.
• The compiled AgentTemplate arrives as KAGENT_CONFIG_JSON: system prompt, model, MCP servers.
• The image is pinned by digest. Tags are rejected.

Three resources wire it up. A Harness says how the agent may run:

apiVersion: kagent.dev/v1alpha3
kind: Harness
metadata:
  name: droid
  namespace: kagent
spec:
  byo: {}
  allowedAgentTemplates:
    selector:
      matchLabels:
        kagent.dev/runtime: droid
  workload:
    image: 172.20.0.8:5000/kagent-droid@sha256:9e99804109f05a51268cfb4639f743a3a42232ccec574542f15426279a58d1f7
    command: ["/usr/local/bin/droid-harness"]
  env:
    - name: DROID_AUTONOMY
      value: high
  substrate:
    workerPoolRef:
      name: kagent-default
    snapshotPolicy:
      location: gs://ate-snapshots/kagent/

An AgentTemplate says what the agent does. Mine is a platform engineering Droid with a short rule set: read-only first, never apply to a live cluster unless asked, validate YAML before handing it back. It references a ModelConfig and two MCP servers (both covered below). The image is the same for every Droid agent you define. Only the template changes.

On the build I ran there is no separate Agent object. An AgentTemplate admitted by a Harness’s allowedAgentTemplates selector is the agent.

Why the Python version never resumed

The first harness was about 300 lines of Python: a2a-sdk 1.2.2 for the gRPC server, droid-sdk 0.5.0 for the agent, one droid Session per kagent conversation. Locally it passed everything. A turn went over A2A, Droid ran shell commands, tool calls streamed back, and a second turn remembered the first.

On the cluster the golden snapshot went Ready. Then every session failed with error resuming actor. The worker logs showed runsc restore exiting cleanly, followed by this:

while waiting for container readyz: readyz cancelled for "kagent" after 4.000628793s
(1507 attempts, last error: Get "http://169.254.17.2:8081/readyz": dial tcp 169.254.17.2:8081: connect: connection refused): context canceled

Substrate checkpoints an actor right after /readyz first passes (the golden snapshot) and again after every turn. The next request restores from that snapshot. Whatever was running at checkpoint time has to come back. My Python process did not, and the router gives a resume 5 seconds before it cancels and retries. So no session ever reached Droid.

I tried the obvious fixes. None worked:

• A dual-stack [::] listener instead of 0.0.0.0.
• A quiet boot, with no subprocess and no outbound connections before the golden snapshot.

Then I isolated it with throwaway probe images, all on the same python:3.12-slim-trixie base:

gRPC was not the cause. Python itself did not come back. Claude Code’s native harness, which is Go, restored in about 0.6s on the same workers. I don’t know why Python fails here, and I haven’t tested newer Substrate builds. On this build, the rule is simple: only a Go process can be alive when the actor is checkpointed.

The design that works: a Go front and droid per turn

The second harness inverts the layout. A static Go binary built on a2a-go serves A2A and readyz, and it is the only long-lived process. Each turn runs one droid exec and streams its JSON events back as A2A updates:

func (e *droidExecutor) args(sessionID, prompt string) []string {
	args := []string{"exec", "-o", "stream-json", "--cwd", e.cfg.Cwd}
	if e.cfg.Autonomy != "" && e.cfg.Autonomy != "off" {
		args = append(args, "--auto", e.cfg.Autonomy)
	}
	if e.cfg.Model != "" {
		args = append(args, "-m", e.cfg.Model)
	}
	if e.cfg.Agent.Instruction != "" {
		args = append(args, "--append-system-prompt", e.cfg.Agent.Instruction)
	}
	if sessionID != "" {
		args = append(args, "-s", sessionID)
	}
	return append(args, prompt)
}

The AgentTemplate maps straight onto droid flags: the system prompt becomes --append-system-prompt, the model becomes -m, and the Harness’s DROID_AUTONOMY becomes --auto. The model needs one translation, because kagent’s claude-haiku-4-5 is claude-haiku-4-5-20251001 in Factory’s catalog. A kagent conversation maps to a droid session ID, and later turns pass it with -s.

droid runs in its own process group, and the group is killed when the turn ends:

cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
// ...
defer func() {
	// Reap droid and anything it spawned, so nothing but this process
	// is left in the Actor when Substrate checkpoints it.
	syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
	// ...
}()

Process memory also did not carry from one turn to the next, so everything that must persist sits on the actor’s /data durable directory: droid’s HOME (its session store), the workspace, and a small map from kagent conversation to droid session. With this design the actor checkpoints in 0.26s and restores in 0.4 to 0.6s.

Giving the agent a vendor key it never sees

Droid authenticates to Factory with Authorization: Bearer <key>. The obvious move is an env var on the Harness. kagent rejects it at compile time:

Compatible=False UnsupportedConfiguration: Harness environment "FACTORY_API_KEY":
arbitrary credentialRef values cannot be injected into HTTP headers;
configure credentials on ModelConfig or RemoteMCPServer

That message is the design. Secrets never enter the actor. Substrate’s egress gateway holds them and swaps a placeholder for the real header on hosts a ModelConfig or RemoteMCPServer declares (kagent’s credential injection notes describe the mechanism). Those same declarations are the egress allowlist. A host nobody declared is unreachable: from inside the sandbox, TLS to it ends in SSL: UNEXPECTED_EOF_WHILE_READING.

So the Factory key goes on an OpenAI-type ModelConfig whose base URL is Factory’s API. OpenAI-type credentials are injected as Authorization: Bearer, which is exactly what droid sends:

apiVersion: kagent.dev/v1alpha3
kind: ModelConfig
metadata:
  name: factory-droid
  namespace: kagent
spec:
  provider: OpenAI
  model: claude-haiku-4-5
  apiKeySecret: factory-api-key
  apiKeySecretKey: FACTORY_API_KEY
  openAI:
    baseUrl: https://api.factory.ai

droid also sends model traffic to a second host, prem.factory.ai. Wildcards are rejected (“credential injection requires an exact DNS hostname”), so that host gets its own declaration, a RemoteMCPServer that exists only to carry egress and a header:

apiVersion: kagent.dev/v1alpha3
kind: RemoteMCPServer
metadata:
  name: factory-prem-egress
  namespace: kagent
spec:
  description: Egress and credential binding for prem.factory.ai (droid model traffic). Not an MCP server.
  protocol: STREAMABLE_HTTP
  url: https://prem.factory.ai/
  headersFrom:
    - name: Authorization
      valueFrom: {type: Secret, name: factory-prem-auth, key: Authorization}

Inside the actor, droid sees the placeholder kagent-credential-injected, which the front copies into FACTORY_API_KEY. The front also leaves *.factory.ai entries out of droid’s MCP config, so droid never tries to use that RemoteMCPServer as a tool. telemetry.factory.ai stays blocked, and droid tolerates it.

That gives a useful property for free. The Droid can reach Factory and the MCP servers in its template, and nothing else, and the key is never in its environment. The same idea runs through secretless AI agents.

Cluster access through read-only MCP

The image ships kubectl and helm, but the sandbox has no kubeconfig, and egress only reaches declared hosts. So kubectl inside the Droid can’t reach any cluster, which is what I want. Cluster access comes from kagent-tools, installed read-only (a get/list/watch ClusterRole without Secrets, plus --read-only) and attached as a RemoteMCPServer named k8s-tools:

helm --kube-context kind-kagent-ent upgrade --install droid-k8s-tools \
  oci://ghcr.io/kagent-dev/tools/helm/kagent-tools --version 0.3.0 \
  -n kagent -f deploy/k8s-tools-values.yaml --wait

One trap: chart 0.3.0 defaults its metrics port to the MCP port, so the Deployment declares a duplicate container port and the install fails. The values file sets tools.metrics.port: 8085.

Asked “List the namespaces you can see”, the Droid answered with the 8 namespaces kubectl get ns shows, through k8s-tools.

Run it

Prerequisites: a cluster running kagent 1.x on Agent Substrate, a Factory API key, and a registry your Substrate workers can pull from. The manifests pin my local kind registry, so push the image and update the digest first.

kubectl -n kagent create secret generic factory-api-key --from-literal=FACTORY_API_KEY=fk-...
kubectl apply -f deploy/factory-modelconfig.yaml -f deploy/factory-egress.yaml \
  -f deploy/k8s-tools-mcp.yaml -f deploy/harness.yaml -f deploy/devops-droid.yaml
kubectl -n kagent port-forward svc/kagent-controller 8083:8083 &
kubectl -n kagent port-forward svc/kagent-ui 8001:8080 &
node scripts/kagent-invoke.mjs devops-droid "List the namespaces you can see"
node scripts/kagent-invoke.mjs devops-droid "Follow-up question" --actor ai-<id from the first call>

The factory-prem-auth Secret holds Bearer <same key>. The README shows how to build it from the first Secret without printing the key. After changing either Secret, restart the gateway’s credential cache:

kubectl -n ate-system rollout restart deploy/k8s-credential-provider deploy/atenet-egress

Troubleshooting a byo agent on Substrate

These cost me the most time, and none of them were Droid problems.

Every agent fails with error resuming actor, Claude Code agents included. Check the router log:

substrate ResumeActor failed ... grpc.message="upstream call failed: SendRequest: connection error: stream closed because of a broken pipe"

After ate-api-server restarts, atenet-router can keep a dead connection to it. Restarting the workers or the kagent controller doesn’t help. Restart the router:

kubectl -n ate-system rollout restart deploy/atenet-router

“no free workers available” while the workers are idle. Failed resumes can leave half-resumed actors holding worker slots, and then every agent in that pool queues. Deleting the worker pods frees them:

kubectl -n kagent delete pod -l ate.dev/worker-pool=kagent-default

ModuleNotFoundError on an image that runs fine under Docker. Substrate does not apply the image’s WORKDIR. Use an absolute command, or install your code as a package.

The agent ignores its tools and model. On the build I ran, KAGENT_CONFIG_JSON arrives flat (model, instruction, http_tools at the top level), not in the runtime/agent shape the OSS docs describe. Parse both.

A turn is rejected as an unknown task. kagent creates the A2A task first and sends its ID with the message. The server has to adopt that ID instead of minting its own.

What’s next

The Droid is read-only on purpose. The next step is human-in-the-loop approval for MCP servers marked requireApproval, through A2A’s input-required state, before anything gets write access to a cluster. A2A task history also lives in the Go front’s memory for now, so kagent’s UI may not show past turns for this agent.

The pattern generalizes beyond Factory. Any CLI agent with a headless JSON mode can run on kagent the same way: a Go front that survives checkpoint, the CLI per turn, state on /data, and credentials declared to the gateway instead of handed to the agent. The code is at github.com/themsquared/kagent-droid.