An agent runtime is the thing that starts, stops, and isolates an agent while it works. kagent puts that on Kubernetes.
These posts cover the harness, the substrate that packs many agents onto few nodes, and what breaks when isolation is shared. Identity for those agents is in Agent Identity.
Start here
- What AI Agent Harness Is Kubernetes-Native? kagent kagent is a Kubernetes-native AI agent harness. What a cloud-native harness is, and how kagent splits harness, template, instance and sandbox.
- kagent Agent Substrate: Thousands of Agents kagent Agent Substrate is built to run thousands of AI agents on tens of pods. I stood it up on kind: snapshot restores, autoscaling, and every gotcha I hit.
- Agent Runtimes Have an Autoscaler, Not a Scheduler Agent runtimes optimize for density and reject when full. That is not scheduling. Three failure modes at capacity, and who gets to call an agent critical.
Every post in this hub
- kagent on Agent Substrate: The Runtime and the Scheduler
kagent runs the agent. Agent Substrate decides where it runs and routes to it. A kind walkthrough of kagent 1.0 on Substrate with a companion repo. - Ghost Actors Pin Your Workers
Ghost actors pin your workers on Agent Substrate: a killed session can hold a slot while the pool looks idle and new chats still get rejected. - The Autoscaler Made It Worse
The autoscaler made it worse on a serial model backend: adding Agent Substrate workers moved the queue downstream and made every turn slower. - CPU Is the Wrong Signal to Autoscale Agents
CPU is the wrong signal to autoscale agents on Agent Substrate: workers are slot-bound and idle at single-digit mCPU. Scale on rejections instead. - The Actors List Lies. Trust the Workers List.
The actors list lies on Agent Substrate: suspended inventory grows without bound and returns partial results. Trust the workers list instead. - Who Runs Next When the Agent Pool Is Full
Who runs next when the agent pool is full is decided by retry races: Agent Substrate rejects, callers retry, and the most aggressive client wins the slot. - Agent Checkpoint to Object Storage
Agent checkpoint to object storage is how Agent Substrate parks RAM and disk. The runtime should decide when to commit. The storage path should own integrity. - Running Agents in Production on Kubernetes
Running agents in production on Kubernetes takes identity, egress, a demand signal, and a kill switch. The runtime you pick does not replace that list. - How Many Agents Per Kubernetes Pod
How many agents per Kubernetes pod depends on which number you mean: live sessions equal workers, registered agents equal snapshots. My laptop ran 18 on two. - Pod Per Agent
Pod per agent is how Kubernetes wants you to run AI agents, and it reserves CPU and memory for processes that are mostly idle. When I stop doing that. - Agent Substrate Day Wrap-Up
Agent Substrate day wrap-up: the 19 topic posts from today plus the field guide hub. Isolation, identity, production, scheduling, and what kubectl cannot show. - Agent Substrate Enterprise
Agent Substrate enterprise is a bar, not a SKU: identity, isolation, scheduling, audit, and a reset path. What I would require before a platform team depends on it. - Agent Substrate Environment
Agent Substrate environment is the higher-level API agents actually want: remote files and process lifecycle on top of sandboxed actors, not CRDs. - Kubernetes Sandbox Platform
A Kubernetes sandbox platform is Agent Substrate without the LLM: dense sandboxes, suspend and resume, and a worker pool that pods do not give you. - Scheduling Agent Sandboxes
Scheduling agent sandboxes is the missing OS layer: Agent Substrate multiplexes actors onto workers but does not decide who runs next when the pool is full. - Scheduling Sandboxes for RL
Scheduling sandboxes for RL multiplexes idle rollouts: each step fans out into sandboxes waiting on an LLM. Suspend during the call, or pay a pod per rollout. - GPUs for Agents on Kubernetes
GPUs for agents on Kubernetes should sit outside Agent Substrate snapshots. GPU memory does not checkpoint with the actor. Keep inference on a shared service. - Suspend-Safe Networking
Suspend-safe networking is the missing piece: suspending an actor today cuts its TCP connections, so an agent waiting on a slow call cannot be parked. - Agent Platform at Scale
An agent platform at scale is not more pods. It is identity, scheduling, and suspend policy for fleets where most sessions are started by other agents. - kubectl Is Lying to You: What a Live Visualizer Taught Me About Agent Substrate
kubectl is lying to you on Agent Substrate: get pods shows quiet workers while actors restore and checkpoint. Talk recap plus four lab lessons. - Agent Substrate Dataplane
Agent Substrate dataplane: ateapi holds actor state, atenet routes, atelet moves snapshots, ateom runs the sandbox. What I watch, and what I restart. - Suspend and Resume Agents
Suspend and resume agents by checkpointing memory and disk to object storage, then restoring on any free worker. What I snapshot, and what I leave out. - Why Agent Substrate Fails
Why Agent Substrate fails, or doesn't: five reasons I thought multiplexed actors on Kubernetes would collapse, and which of those risks I still treat as real. - kubectl Is Lying to You
kubectl is lying to you on Agent Substrate: get pods shows quiet workers while agents restore, queue, and checkpoint. Watch the workers list instead. - Running Agent Substrate in Production
Running Agent Substrate in production means real snapshot storage, a durable control plane, a secrets path, and a sandbox class you can defend. - Agent Sandbox vs Agent Substrate
Agent sandbox vs Agent Substrate: run agents as pods, isolate each one in a sandbox, or multiplex snapshot-backed actors onto a worker pool. How I choose. - What Is Agent Substrate?
What is Agent Substrate: a Kubernetes runtime that runs AI agents as snapshot-backed actors on a small worker pool instead of one always-on pod per agent. - Agent Substrate: the Field Guide
Agent Substrate field guide: how the runtime packs AI agents onto Kubernetes, what to scale on, and the isolation, identity, and scheduling it leaves to you. - The kagent Harness: Declarative Agents, Memory and Compaction
kagent's built-in harness runs a Go engine that reads your AgentTemplate: why it's the default, how memory and compaction work, and what you get that Claude Code, Codex and BYO don't. - kagent BYO Harness: Running Factory Droid as an A2A Agent
What kagent's byo harness requires, and how kagent-droid runs Factory Droid behind it: a Go A2A front, one droid exec per turn, session continuity on /data, and read-only k8s-tools. - Does Agent Substrate's Checkpoint Survive a Held File Lock?
I tried to break Agent Substrate's checkpoint with a held file lock. It survived: gVisor virtualizes flock() inside its own sentry, not the host kernel. - kagent Harnesses Compared: kagent vs Claude Code vs Codex vs BYO
The four kagent 1.0 harnesses side by side: what each runs, which model providers it takes, what it drops from an AgentTemplate, and how to pick one. - kagent Codex Harness: Run OpenAI Codex on Kubernetes
How to run OpenAI Codex as a kagent 1.0 harness: the four resources, the ModelConfig rules, MCP and env restrictions, and how to debug a Compatible=False agent. - How AI Agents Sharing a Sandbox Leak Each Other's Data
A finance AI agent leaked bank balances into company Slack by sharing compute with another agent. Reproduced in Docker, then fixed with separate sandboxes. - What AI Agent Harness Is Kubernetes-Native? kagent
kagent is a Kubernetes-native AI agent harness. What a cloud-native harness is, and how kagent splits harness, template, instance and sandbox. - Agent Runtimes Have an Autoscaler, Not a Scheduler
Agent runtimes optimize for density and reject when full. That is not scheduling. Three failure modes at capacity, and who gets to call an agent critical. - Regression Tests for kagent Agents with agentevals
kagent records agent sessions as ADK events. agentevals scores chat-message trajectories. I built the bridge, and a suite that catches an ungrounded agent. - kagent Claude Code Harness on Kubernetes (Updated for v1alpha3)
Run Claude Code as a kagent harness: current Harness + AgentTemplate + Agent YAML, the MCP all-tools gotcha, plus the v1.0.0-alpha1 history (AgentInstance, checkpoints, golden snapshots). - Substrate Scope: a Live Visualizer for Agent Substrate
Substrate Scope is an open source live visualizer for Agent Substrate: worker bays, restore queues, snapshot storage, telemetry, and per-agent activity. - Revenant: A Gateway-Native, Always-On Agent Runtime
Revenant is an always-on agent runtime in Rust where an AI gateway owns every key, budget, and failover, and a hard liveness rule means no turn goes silent. - kagent Agent Substrate: Thousands of Agents
kagent Agent Substrate is built to run thousands of AI agents on tens of pods. I stood it up on kind: snapshot restores, autoscaling, and every gotcha I hit.