# kagent on Agent Substrate: The Runtime and the Scheduler

> kagent runs the agent. Agent Substrate decides where it runs and routes to it. A kind walkthrough of kagent 1.0 on Substrate with a companion repo.

- Canonical URL: https://webofmike.com/kagent-on-agent-substrate/
- Author: Mike Moore (https://webofmike.com/about/)
- Published: 2026-11-15
- Last modified: 2026-11-15
- Tags: AI Agents, Kubernetes, Platform Engineering, Tutorials
- Cite as: Mike Moore, "kagent on Agent Substrate: The Runtime and the Scheduler", Web of Mike (webofmike.com), 2026-11-15. https://webofmike.com/kagent-on-agent-substrate/


<!-- ART: two stacked layers in a data center cutaway, the upper layer a conductor's desk with a score and a baton, the lower layer a switching yard with a dispatcher routing small train cars onto a few waiting platforms. -->

kagent is the runtime: it defines each agent as a Kubernetes resource, compiles it, runs the agent loop inside a Harness engine, and owns the conversation. Agent Substrate is not the runtime. It is the scheduling and routing layer under kagent: it decides which worker an agent's actor lands on, suspends and resumes that actor between turns, and routes every request to wherever the actor currently is. Keep those two jobs apart and a kagent 1.0 install is easy to reason about. Blur them and you debug the wrong system.

This is the long version, written in the wake of KubeCon North America 2026 (Salt Lake City, November 9 to 12, which added an [AI Inference + Agentic track](https://www.cncf.io/announcements/2026/08/10/cncf-reveals-kubecon-cloudnativecon-north-america-2026-schedule-adds-new-ai-inference-agentic-track/)). Everything here is runnable. The companion repo, [themsquared/kagent-on-substrate](https://github.com/themsquared/kagent-on-substrate), has the kind config, the install config, two WorkerPools, two Harnesses, one AgentTemplate, two Agents, a demo script, and a `make validate` target that checks every manifest against the real CRDs. By the end you will have one agent definition running on two placement domains, and you will be able to point at each object and say which project owns it.

If you want the background first, start with [what Agent Substrate is](/what-is-agent-substrate/). If you want the argument for why the scheduling half matters, read [agent runtimes have an autoscaler, not a scheduler](/agent-runtimes-need-a-scheduler/). This post is the hands-on companion to both.

## Is Agent Substrate a runtime?

For an agent author, no. The runtime is whatever executes your agent's loop: reads the system prompt, calls the model, calls tools, keeps the conversation. In kagent 1.0 that is the engine a Harness selects. The [Harness docs](https://kagent.dev/docs/kagent/1.x/agents/agent-harness/) list the choices: kagent's own engine (`spec.kagent`), Codex, Claude, or your own image (`byo`). Agent Substrate never sees a prompt. It runs a sandboxed workload and moves it around.

The word "runtime" is used loosely in the docs, so here is the honest map of who says what:

- Solo's [What is kagent](https://docs.solo.io/kagent/latest/about/what-is-kagent/) page calls Solo Enterprise for kagent "a Kubernetes-native runtime for the AI agents". That is the agent runtime.
- The kagent docs glossary describes Agent Substrate as "the runtime that kagent runs agents on", and the Substrate README calls it ["a low-opinion runtime, not an agent SDK"](https://github.com/kagent-dev/substrate/blob/v1.0.1/README.md) for workloads that "do not have to be AI agents". That is a compute runtime: it runs OCI workloads in sandboxes.
- Both are true at different layers. This post uses "runtime" only for the first meaning, and says "scheduling and routing layer" for the second.

There is a simple test for which is which: the dependency only goes one way. kagent 1.0 runs every agent on Agent Substrate, per the [install guide](https://kagent.dev/docs/kagent/1.x/setup/installation/). Substrate will run any workload you give it with no kagent in sight. Delete Substrate and kagent has nowhere to run. Delete kagent and Substrate still works.

### What is the common mistake?

Treating Substrate as the thing you deploy your agent to. It shows up in four forms, and each one sends you to the wrong tool:

| The mistake | Why it goes wrong | What to do instead |
|---|---|---|
| "Deploy the agent to Substrate" | You deploy an `Agent` to kagent. The kagent controller compiles it into an ActorTemplate for Substrate | Author Harness, AgentTemplate, Agent. Never hand-write the ActorTemplate |
| Looking for agent config in `kubectl ate` | Substrate lists workers and actors. It has no agent, prompt, or tool concept | Use `kagent agent get` and `kubectl get agent` for the agent. Use `kubectl ate` for capacity and placement |
| Expecting kagent to pick the worker | kagent names a pool through the Harness. Substrate picks the worker inside it | Size and label pools. Do not look for a worker choice in any kagent resource |
| Expecting Substrate to enforce who runs next when the pool is full | The docs describe parking, not priority. See below | Isolate with pools, and put policy above Substrate |

The rest of the post builds the split into real files.

## Who owns what in a kagent 1.0 install?

Six things matter, and they split cleanly:

| Object | Kind | Owner | What it is |
|---|---|---|---|
| `Harness` | CRD, `api.kagent.dev/v1alpha3` | kagent | The engine that runs the loop, plus which pool and snapshot location it may use |
| `AgentTemplate` | CRD, `api.kagent.dev/v1alpha3` | kagent | Model, system prompt, tools. Portable behavior, no placement |
| `Agent` | CRD, `api.kagent.dev/v1alpha3` | kagent | Pairs one AgentTemplate with one Harness |
| `Session` | gRPC and PostgreSQL object, not a CRD | kagent | One conversation with an Agent. Pins one compiled revision |
| `WorkerPool` | CRD, `ate.dev/v1` | Substrate | Capacity: how many workers, which image, which sandbox class, how big |
| `Actor` | Record in Substrate's control plane | Substrate | The suspendable unit a Session runs in. Named `session-<id>` |

Sources: the kagent [core concepts](https://kagent.dev/docs/kagent/1.x/about/core-concepts/), the kagent v1.0.1 [runtime and lifecycle notes](https://github.com/kagent-dev/kagent/blob/v1.0.1/docs/architecture/runtime-and-lifecycle.md), and the actor naming in the [kagent source](https://github.com/kagent-dev/kagent/blob/v1.0.1/go/core/internal/substrate/client.go).

Here is the picture to hold in your head:

```
                       caller (A2A client, CLI, UI)
                                  |
                     +------------v-------------+
   kagent            |  Agent = AgentTemplate   |   what the agent is
   the runtime       |        + Harness         |   the loop, tools, Session,
                     |  Session, MCP tools      |   the /agents/{ns}/{name} front door
                     +------------+-------------+
                                  |   Harness.spec.substrate.workerPoolRef
                     +------------v-------------+
   Agent Substrate   |  router: find the actor  |   where it runs
   scheduling and    |  scheduler: pick worker  |   which worker, resume, suspend,
   routing           |  WorkerPool, snapshots   |   park the request when full
                     +--------------------------+
```

One note on the 0.x line. The kagent 1.0 API group `api.kagent.dev` is unrelated to the old `kagent.dev` group, and there is no in-place upgrade between them, per the [install guide](https://kagent.dev/docs/kagent/1.x/setup/installation/). If you have a 0.x cluster, treat this as a fresh install.

## What do you need before you start?

- Docker and [kind](https://kind.sigs.k8s.io/) v0.32.0 or later. Kubernetes 1.37 or later is required because Agent Substrate depends on the `PodCertificateRequest` and `ClusterTrustBundle` APIs, which are generally available in 1.37 ([install guide](https://kagent.dev/docs/kagent/1.x/setup/installation/)).
- `kubectl`, `helm` 3, and `jq`.
- The `kagent` CLI at the same version as the release you install. The repo pins v1.0.1.
- `OPENAI_API_KEY` exported in your shell. The installer reads it and stores it in a cluster Secret. It never goes in a file in the repo.

Clone the companion repo and run `make tools` to check the CLIs. Every step below is a `make` target.

## How do you install kagent and Substrate on kind?

Two commands:

```bash
make cluster
export OPENAI_API_KEY=...   # your key
make install
```

`make cluster` creates a single-node kind cluster on `kindest/node:v1.37.0` from [cluster/kind-config.yaml](https://github.com/themsquared/kagent-on-substrate/blob/main/cluster/kind-config.yaml). The kagent docs say no API server changes are needed.

`make install` runs `kagent install`. In one pass it installs the CRDs, Agent Substrate into `ate-system`, the Substrate PodCertificate controller, a development PostgreSQL, and kagent. kagent and Substrate authenticate to each other with Pod Certificates instead of passwords. The development database runs the stock `postgres` image with no `pgvector`, which long-term memory needs, so it is for evaluation only.

The installer takes no Helm flags. Chart values reach it through a CLI config file, which the repo keeps in [install/kagent-config.yaml](https://github.com/themsquared/kagent-on-substrate/blob/main/install/kagent-config.yaml):

```yaml
install:
  kagent:
    valuesFiles: [kagent-values.yaml]
  substrate:
    valuesFiles: [substrate-values.yaml]
```

The `install.kagent.valuesFiles` field is in the [install guide](https://kagent.dev/docs/kagent/1.x/setup/installation/). The `install.substrate` block is the same mechanism for the Substrate release, documented in the [kagent chart README](https://github.com/kagent-dev/kagent/blob/v1.0.1/helm/README.md). The two files do two small things:

- `kagent-values.yaml` makes the bundled `kagent-tools` MCP server read-only, with a read-only ClusterRole instead of the chart's default cluster-admin. The demo agent triages pods. It should not be able to change them.
- `substrate-values.yaml` sets `atenetRouter.requestParking.budget: "60s"`. That is the chart default. It is in the file so the knob that controls pool-full behavior is visible in the repo. More on it in a moment.

`kagent install --dry-run` renders both into the real chart values, which is how the repo checks that the files reach the charts. When the install finishes you have the stock `kagent-default` WorkerPool. Check it:

```bash
kubectl get workerpools -n kagent
```

Then install Substrate's CLI, `kubectl-ate`, from the release:

```bash
make ate-cli
kubectl ate get workers
```

Look at what that command lists: workers and actors. Not agents. That is the split showing up in the tooling.

## How do you define the pools Substrate schedules onto?

A [WorkerPool](https://github.com/kagent-dev/substrate/blob/v1.0.1/docs/reference/api-guide.md) is a placement domain: a fleet of pre-warmed worker pods that actors get multiplexed onto. It is Substrate's resource, in `ate.dev/v1`. The repo defines two, in [manifests/10-workerpools.yaml](https://github.com/themsquared/kagent-on-substrate/blob/main/manifests/10-workerpools.yaml). Here is the small one:

```yaml
apiVersion: ate.dev/v1
kind: WorkerPool
metadata:
  name: batch
  namespace: kagent
  labels:
    kagent.dev/worker-pool-name: batch
    kagent.dev/worker-pool-namespace: kagent
    tier: batch
spec:
  replicas: 1
  workerImage: ${WORKER_IMAGE}
  sandboxClasses:
    - name: gvisor
      configRef:
        name: gvisor-default
  template:
    serviceAccountName: kagent-default
    resources:
      requests:
        cpu: 250m
        memory: 512Mi
      limits:
        cpu: "2"
        memory: 3Gi
```

The `interactive` pool is the same shape with `replicas: 2` and limits of 4 CPU and 6Gi. Three details carry the lesson.

**The limits are the capacity envelope.** Per the Substrate [API guide](https://github.com/kagent-dev/substrate/blob/v1.0.1/docs/reference/api-guide.md), the pool's `resources.limits` set each worker's capacity, the budget its actors share. The scheduler only places an actor on a worker whose remaining capacity covers that actor's declared limits. Who declares the actor's limits? kagent does. The chart's `config.runtimeDefaults` sets limits for each agent runtime, 1 CPU and 1Gi by default, and the chart comments say "Substrate requires both". So the `batch` pool above has room for roughly two default-sized actors at once. I made it small on purpose. A burst of batch work saturates it fast, and that is the behavior I want to look at.

**Two labels are mandatory.** Every pool that backs a Harness must carry both `kagent.dev/worker-pool-name` and `kagent.dev/worker-pool-namespace`. kagent's compiler selects workers with those two labels ([configuration and compilation](https://github.com/kagent-dev/kagent/blob/v1.0.1/docs/architecture/configuration-and-compilation.md)). The same page has a sharp edge worth knowing: Substrate matches worker labels without checking them against the pool's namespace, so placement stays inside a namespace only while every pool's labels name the pool's own namespace. A pool with the wrong namespace label is a pool that can receive another namespace's actors. The labels are a contract, so treat them like one.

**The pool also owns the sandbox class.** `sandboxClasses` picks gVisor here, and the compilation notes say the WorkerPool owns runtime selection. If you change a pool's class, existing Sessions stay pinned to their revisions, so use a separate pool when old revisions must keep the old runtime.

Notice what this file does not contain: any agent. A pool knows nothing about prompts, models, or tools.

## How does kagent choose a pool without choosing a worker?

Through the Harness. This is the only place the two projects touch in your manifests, and it is the cleanest illustration of the division, so it is worth reading slowly. From [manifests/20-harnesses.yaml](https://github.com/themsquared/kagent-on-substrate/blob/main/manifests/20-harnesses.yaml):

```yaml
apiVersion: api.kagent.dev/v1alpha3
kind: Harness
metadata:
  name: kagent-batch
  namespace: kagent
spec:
  kagent: {}
  substrate:
    workerPoolRef:
      name: batch
    snapshotPolicy:
      location: s3://ate-snapshots/kagent/
```

`spec.kagent: {}` selects kagent's own engine. That is the runtime. `spec.substrate` says which pool its actors may run on and where their snapshots go. The location is the bucket the Substrate chart creates in its bundled object store, as in the [first agent guide](https://kagent.dev/docs/kagent/1.x/get-started/your-first-agent/). The sibling `kagent-interactive` Harness is identical except for `workerPoolRef.name: interactive`.

What happens next is the part that is easy to miss. kagent does not tell Substrate "put this on pool batch". It compiles the reference into an ActorTemplate with a worker selector on the two pool labels, and Substrate evaluates the selector. The Substrate API guide describes the `workerSelector` as gating "which WorkerPools actors from this template may use, by matching against each pool's labels". So the handoff is a label match. kagent expresses an intent. Substrate resolves it to a worker.

I left `spec.workload` out, so each Harness runs the release's built-in kagent runtime image. The Harness CRD marks `workload` as an override that only BYO requires. The first agent guide pins `workload.image` by digest in its example. If you want an image you can audit, pin it.

Two Harnesses, one engine, two pools. Same runtime, different compute policy. That is the whole trick.

## How do you define the agent once and run it on both pools?

The AgentTemplate is portable behavior: model, prompt, tools. It says nothing about where it runs. From [manifests/30-agenttemplate.yaml](https://github.com/themsquared/kagent-on-substrate/blob/main/manifests/30-agenttemplate.yaml):

```yaml
apiVersion: api.kagent.dev/v1alpha3
kind: AgentTemplate
metadata:
  name: pod-triage
  namespace: kagent
spec:
  description: Read-only Kubernetes triage. Explains why a pod is unhealthy and what to check next.
  modelConfig:
    name: default-model-config
  systemPrompt: |
    You are a Kubernetes triage assistant. You can only read cluster state.
    Use the tools to look at pods, events and logs before you answer.
    Answer in at most five lines: what is wrong, the evidence, and the next command to run.
    Never suggest a change you have not backed with evidence from a tool call.
  tools:
    - mcp:
        server:
          kind: RemoteMCPServer
          name: kagent-tool-server
        tools:
          - k8s_get_resources
          - k8s_describe_resource
          - k8s_get_events
          - k8s_get_pod_logs
```

`default-model-config` is the ModelConfig the installer created from your `OPENAI_API_KEY`. `kagent-tool-server` is the RemoteMCPServer the chart registers for the bundled tools. The four tool names come from kagent-tools 0.3.0; after install, `kubectl -n kagent get remotemcpserver kagent-tool-server -o jsonpath='{.status.discoveredTools[*].name}'` lists what your server really offers. The repo's `examples/` folder has a second ModelConfig and a custom RemoteMCPServer if you want to swap either.

Then the pairing, in [manifests/40-agents.yaml](https://github.com/themsquared/kagent-on-substrate/blob/main/manifests/40-agents.yaml):

```yaml
apiVersion: api.kagent.dev/v1alpha3
kind: Agent
metadata:
  name: triage-batch
  namespace: kagent
spec:
  template:
    ref:
      name: pod-triage
  harnessRef:
    name: kagent-batch
```

`triage-interactive` is the same with `harnessRef.name: kagent-interactive`. Apply everything:

```bash
make apply
```

An Agent is not `Ready` immediately. Per the first agent guide, kagent compiles an immutable revision and waits for a golden snapshot of the runtime, so `READY` can read `False` for about a minute. `make apply` waits for it. The Agent reports four conditions, `Accepted`, `ResolvedRefs`, `Compatible`, and `Ready`. If it sticks, `kubectl get agent triage-batch -n kagent -o jsonpath='{.status.conditions}'` says where.

Look back at what you wrote. Callers address `/agents/kagent/triage-batch` on the kagent A2A gateway ([A2A gateway notes](https://github.com/kagent-dev/kagent/blob/v1.0.1/docs/architecture/a2a-gateway.md)). They never name a pool, a worker, or an actor. Placement is invisible to the caller by design.

## What happens to one request, end to end?

Here is a single message, with the owner of each step marked. The sequence comes from the kagent [suspend and resume](https://kagent.dev/docs/kagent/1.x/about/substrate-runtime/suspend-and-resume/) page and the Substrate architecture and parking docs.

1. **Create a Session (kagent).** `kagent agent session create --agent triage-interactive` records a Session in kagent's PostgreSQL, pins the Agent's latest ready revision, and creates the Session's Actor. The runtime notes say the Actor starts suspended, so it holds no worker yet.
2. **Send a message (kagent).** The CLI sends it over A2A. The conversation's context identifier is the Session ID. The kagent controller handles it.
3. **Hand to the router (kagent to Substrate).** The controller addresses the Actor by name, `session-<id>`. The traffic goes through the Substrate router. The install passes `atenetRouter.ingressAllowedSPIFFEIDs` so the controller's identity is allowed in.
4. **Resume (Substrate).** The actor is suspended, so the router asks the control plane to resume it. The scheduler considers eligible workers: active, selector match, right sandbox class, room for the actor's limits. Among those it samples two at random and takes the less loaded one ([Learn Agent Substrate: workers](https://learn.agentsubstrate.dev/components/workers/)). That is placement.
5. **Restore (Substrate).** The worker starts the sandbox and restores the Actor from its last snapshot. The restore is `Data` scope: kagent sets that on every ActorTemplate it generates, so a resume starts new containers from the image and restores the durable `/data` directory. Process memory is not preserved.
6. **Run the loop (kagent).** The Harness engine in the actor takes the message: it calls the model, calls MCP tools, and appends to the transcript. Outbound calls go through Substrate's egress gateway, which injects model and MCP credentials, so the agent never holds the provider key ([What is kagent](https://docs.solo.io/kagent/latest/about/what-is-kagent/)).
7. **Suspend at the turn boundary (kagent asks, Substrate does).** When the turn ends, or stops to wait on a person, kagent suspends the Actor. Substrate writes the snapshot and releases the capacity the Actor held. The Session stays `READY` the whole time.
8. **Next message resumes it.** Possibly on a different worker. Substrate resolves the actor's address on every request, so the name stays stable ([suspend and resume](https://kagent.dev/docs/kagent/1.x/about/substrate-runtime/suspend-and-resume/)).

Count the owners. Steps 1, 2, 6 and the decision in 7 are kagent. Steps 4, 5, 7's mechanics and 8's routing are Substrate. Step 3 is the seam. The docs also state the cost plainly: because kagent suspends at every turn boundary, every message to a Session pays for one resume.

## How do you watch Substrate do its job?

Run the demo:

```bash
make demo
```

[scripts/demo.sh](https://github.com/themsquared/kagent-on-substrate/blob/main/scripts/demo.sh) prints the real output of each command. It creates a Session on the interactive Agent, sends a turn, shows the pools and the actors, sends a follow-up, then fires a burst of Sessions at the small `batch` pool while it sends one more message to the interactive Session. It asserts nothing about timing, because timing is yours to read.

Two commands carry the story:

```bash
kubectl ate get workers
kubectl ate get actors -a kagent
```

The workers list shows each worker's actors and its CPU and memory, allocated against capacity. Read occupancy from there. The actors list is history. Every finished Session leaves a suspended actor behind, so it grows without bound and is not live inventory. I hit that the hard way, and wrote it up in [the actors list lies](/substrate-actors-list-lies/). Actors live in an atespace named after the Agent's namespace, which is why the `-a kagent` flag is there.

For metrics, the pool exports `ate.workerpool.workers`, `ate.workerpool.desired_workers`, and `ate.workerpool.ready_workers`, and the router exports `atenet.router.parking.active`, `atenet.router.parking.wait.duration`, and `atenet.router.parking.rejected` ([request parking](https://github.com/kagent-dev/substrate/blob/v1.0.1/docs/concepts/request-parking.md)). The WorkerPool also has a scale subresource, so a Kubernetes HPA can scale it. The enterprise and OSS feature tables both list pool autoscaling through HPA.

If you want to go deeper on reading Substrate state, [kubectl is lying to you](/kubectl-is-lying-to-you-watch-agent-substrate/) covers why `kubectl get pods` cannot tell you what a worker is doing.

## What happens when the pool is full?

This is where the two layers earn their keep, and where it is easiest to credit the wrong one. Everything below is Substrate behavior. kagent just sends a message.

When a message arrives for a suspended actor and no worker has room, the resume comes back `ResourceExhausted`. On 1.0 the router does not fail that request. It parks it. Per the [request parking doc](https://github.com/kagent-dev/substrate/blob/v1.0.1/docs/concepts/request-parking.md):

- The router treats `ResourceExhausted` and `Unavailable` as retryable and keeps retrying the resume with exponential backoff until the actor is routable or the park budget runs out.
- The retry shape is a 100 millisecond first delay, a 1.1 multiplier, and 0.1 jitter, "to de-synchronize parked requests".
- The budget bounds retries, not a committed resume. An attempt already in flight is not canceled, so a restore that overshoots the budget is served late rather than failed.
- If the budget runs out, the caller gets a `503` with the capacity error.
- Parked requests sit in a lot of fixed size, 1024 by default. A full lot sheds with a `503` "router at capacity".
- Concurrent requests for the same actor share one in-flight resume, so a hot actor takes many lot slots but one control-plane call.

The budget has two defaults and they disagree. The router binary's `--parked-request-budget` defaults to `5s`. The Substrate chart sets `atenetRouter.requestParking.budget` to `60s`, which is what `helm show values` prints for the 1.0.0 chart. Since kagent installs Substrate from the chart, your effective budget is the chart's 60 seconds unless you change it. That is why the repo writes it into `substrate-values.yaml`: a knob this consequential should not be implicit.

This is also the part of the story that changed since my earlier posts, so I will be direct about it. In [Who Runs Next When the Agent Pool Is Full](/who-runs-next-when-the-agent-pool-is-full/) I described reject and retry. The 1.0 parking doc says the router "previously mapped that straight to an HTTP 503" and now holds the request first, so check which behavior your build has.

Now the part that did not change. Parked requests retry independently, with backoff and jitter. The docs do not describe an ordering among them, and they do not describe priority. So a pool-full 1.0 system rides out a momentary blip without a caller seeing an error, which is a real improvement. It does not decide who runs next. I would not assume first in, first out.

This is why the repo has two pools. The `interactive` and `batch` pools are separate placement domains. A burst that saturates `batch` parks in `batch`'s queue and cannot take an `interactive` slot. That is the isolation tool the docs actually hand you, and the demo is built to show it.

One thing I cannot tell you from the docs is what the kagent controller does when the router returns the `503` after the budget runs out. The sources I read do not say. If you are sizing a production pool, test that path.

## What does Substrate not do for you?

The public docs I read describe placement, snapshots, suspend and resume, routing, parking, and egress. They do not describe:

- priority classes or preemption among actors
- per-team quotas or fair queueing
- an ordering guarantee among parked requests
- any admission policy that looks at which agent or which user is asking

I wrote up the case for why that gap matters in [Scheduling Agent Sandboxes](/scheduling-agent-sandboxes/), and the failure mode in [Who Runs Next When the Agent Pool Is Full](/who-runs-next-when-the-agent-pool-is-full/). I will not repeat the argument. Here is the narrower point for this post: this gap is not a bug in the split. It is what the split looks like. A layer that multiplexes opaque workloads onto workers should not know that one workload is a billing agent and another is a demo. Where that knowledge lives is a design question.

My take, and it is an opinion rather than a documented feature: kagent knows about Agents, Sessions and users, so admission policy that cares about who is asking fits above Substrate, at the Session boundary, with pools as the coarse isolation underneath. Nothing in the docs I read does this today. If you need priority now, pools are the lever.

## What changed since my 0.x lab?

My [August lab](/kagent-agent-substrate/) ran Agent Substrate under the older kagent. The 1.0 install changes enough that it is worth listing what moved:

- **The API.** An agent is now three resources, Harness, AgentTemplate and Agent, in `api.kagent.dev/v1alpha3`. A Session is a gRPC and PostgreSQL object, not a CRD. The old `kagent.dev` group has no upgrade path.
- **Placement.** The 1.0 docs describe a capacity-aware scheduler: a free actor slot plus a CPU and memory check, then power of two choices ([Learn Agent Substrate](https://learn.agentsubstrate.dev/components/workers/)). My lab post described a worker as hosting one actor at a time. The Substrate architecture docs and API guide describe a worker as a multi-actor host with a capacity envelope, while one line in the kagent glossary still says a Worker hosts "at most one Actor at a time". I have not measured this on 1.0. Check the actors column of `kubectl ate get workers` on your own cluster before you trust either claim.
- **Pool-full behavior.** Reject became park, as above.
- **Suspension cadence.** kagent suspends at every turn boundary, not after idle time. The cost is one resume per message.
- **The harness choice.** The engine is now a Harness field, not a deployment fact. My [harness comparison](/kagent-harnesses-compared/) and the [native harness post](/kagent-native-harness-declarative-agents/) cover that side.

## Does this work on the enterprise build?

The Solo Enterprise for kagent docs describe the same model: Harness, AgentTemplate and Agent on Agent Substrate, with the same `api.kagent.dev/v1alpha3` APIs. The [enterprise install guide](https://docs.solo.io/kagent/latest/setup/installation/) replaces the install step in this walkthrough. After it, apply the manifests the same way.

The public feature table lists what the enterprise distribution adds on top: single sign-on, catalog access control, tenancy controls, telemetry and analytics, and a management console with Substrate lifecycle visibility. None of that changes the split in this post. kagent is still the runtime and Substrate is still the scheduling and routing layer underneath. For production shapes, [running Agent Substrate in production](/running-agent-substrate-in-production/) covers snapshot storage, the durable control plane, and the sandbox class.

Two things to check on an enterprise build, because I did not run them there: the Harness runtime image (the [enterprise first agent guide](https://docs.solo.io/kagent/latest/get-started/your-first-agent/) pins `workload.image` by digest, and the repo omits `workload`), and the stock pool's name and ServiceAccount (the repo reuses the OSS chart's `kagent-default`).

## What did I not verify?

Honesty about the edges is the point of a post like this, so here is the list.

- **The demo has not been run end to end by the author.** The manifests are verified. `make validate` starts a throwaway kube-apiserver v1.37.0 with the real kagent 1.0.1 and Substrate 1.0.0 CRDs and runs `kubectl apply --dry-run=server` on every manifest. That checks the schemas and the CEL rules, which a schema-only linter does not. Six negative manifests, such as a Harness with two engines or an image pinned by tag, are rejected as they should be. But the repo's author had no Docker for a live kind run. The commands are the documented ones, the demo prints real output, and the first live run is yours. If something differs, open an issue on the repo.
- **Behavior descriptions come from docs and source, not from measurement.** Placement, parking, suspend and resume, and the budget numbers are as the linked docs state them.
- **The kagent-tools tool names** were checked against kagent-tools 0.3.0. Check `status.discoveredTools` on yours.
- **Reusing `kagent-default` as the pool ServiceAccount** works by design in the chart, but I have not run it with two extra pools.
- **Per-Harness snapshot locations.** Both Harnesses use the documented location. I did not test distinct prefixes.
- **Worker capacity.** The one-actor versus many-actors conflict above is real in the docs.
- **Router behavior on the enterprise build.** The parking numbers are from the open source chart.

## What are the common questions about kagent on Agent Substrate?

### Is Agent Substrate a runtime for kagent agents?

Not in the sense that matters to an agent author. kagent's Harness engine runs the loop: the prompt, the model calls, the tool calls, the conversation. Agent Substrate is the compute layer under it. It places actors on workers, suspends and resumes them from snapshots, and routes each request to the right actor. It does not know what an agent loop is. Substrate's own README calls it a low-opinion runtime for any workload, so the word is used loosely in the docs; the job split is not.

### How does kagent choose which worker an agent runs on?

It does not. A kagent Harness names a WorkerPool in `spec.substrate.workerPoolRef`. kagent compiles that into an ActorTemplate whose worker selector matches two pool labels, `kagent.dev/worker-pool-name` and `kagent.dev/worker-pool-namespace`. Agent Substrate's scheduler then picks a worker inside the matching pools: it needs a free actor slot and enough CPU and memory, and it samples two eligible workers at random and takes the less loaded one.

### What happens when the WorkerPool is full on kagent 1.0?

The Agent Substrate router parks the request and retries the resume with exponential backoff and jitter until a worker has room or the park budget runs out. The chart default for `atenetRouter.requestParking.budget` is `60s`, while the router binary's own flag default is `5s`. If the budget runs out the caller gets a `503`. The docs describe no priority or ordering among parked requests.

### Does Agent Substrate provide priority or fairness between teams?

The public docs I read describe placement, suspend and resume, routing, and request parking. They do not describe priority classes, per-team quotas, or fair queueing. Separate WorkerPools behind separate Harnesses are the isolation tool the docs give you. Anything finer is policy you would have to add above Substrate.

### Can I run kagent on Agent Substrate on a laptop?

Yes, on kind. kagent 1.0 needs Kubernetes 1.37 or later and kind v0.32.0 or later, because Agent Substrate depends on the `PodCertificateRequest` and `ClusterTrustBundle` APIs. The `kagent install` command installs Substrate, a development PostgreSQL, and kagent in order. The companion repo has the kind config, the install config, the manifests, a demo script, and a validation script.

### Does this work with the enterprise build of kagent?

The Solo Enterprise for kagent docs describe the same Harness, AgentTemplate, and Agent model on the same Agent Substrate layer. The install path differs, so follow the enterprise install guide for that step and apply the same manifests after it. I have not run the repo against an enterprise installation, and the enterprise docs pin the Harness runtime image by digest where the repo omits it.

## Where to go from here

Run the repo, then change one thing and watch which project reacts. Edit the AgentTemplate prompt, and kagent compiles a new revision while Substrate does nothing. Change a pool's `replicas`, and Substrate reacts while kagent does nothing. Point a Harness at the other pool, and the Agent's placement moves while its behavior stays. If you can predict which half responds, you have the split.

The code is at [themsquared/kagent-on-substrate](https://github.com/themsquared/kagent-on-substrate). The deep background is in [what Agent Substrate is](/what-is-agent-substrate/), [Substrate in production](/running-agent-substrate-in-production/), and [the scheduler argument](/agent-runtimes-need-a-scheduler/).

