SPIFFE Workload Identity for AI Agents, End to End

I built a demo where an AI agent calls a model and nothing in the path holds a certificate file. Not the agent, not the gateway, not the model upstream. Every identity is issued at runtime by SPIRE, rotates on its own, and is verified on the TLS handshake rather than read out of a header. The gateway’s authorization policy is written against a SPIFFE ID. It all runs on docker compose. The code is at themsquared/agent-identity-spiffe.

Yesterday I wrote about agents that hold no LLM credential, where the gateway holds the provider key and the agent authenticates with a short-lived JWT from a local issuer. That issuer was the weak part of the design. I wrote it myself, it signed whatever it was asked to sign, and the claim it put in the token (team: research) was an assertion nobody checked. This post replaces it with the real thing.

A bearer token is the wrong primitive for agent identity

The normal way to authenticate an agent to a gateway is a bearer token: an API key, a static JWT, something the agent presents and the gateway believes. The trouble is structural. A bearer token is a thing that can be copied, so it has to be stored, and wherever it is stored is what an attacker goes for. It lands in an environment variable, gets logged by an HTTP client with verbose tracing on, and shows up in a crash dump. The compromised dependency that reads it gets everything that token can do, for as long as the token lives.

Agent workloads make this worse in two specific ways. They run a lot of third-party code by design, since the whole value proposition is calling tools and libraries on your behalf. And they are increasingly ephemeral, which means the operational pressure is toward long-lived credentials baked into an image, because nobody wants to rotate a secret across a fleet that recreates itself constantly.

SPIFFE takes a different position: stop giving workloads secrets. A workload asks the local Workload API who it is, and gets back an X.509 SVID whose issuance was conditioned on attested properties of the workload itself. There is nothing to copy into a config file because nothing was ever written down. The certificate lives minutes, and renewal is a background stream rather than an operational event.

What agentgateway v1.5.0 added

The gateway is the part that was missing. agentgateway could already terminate mTLS, but only from a static cert and key on disk, which reintroduces exactly the file you were trying to eliminate. v1.5.0 added the Workload API as an identity source, and three things follow from it:

1. The gateway fetches its own SVID and trust bundle from the Workload API and rotates them automatically.
2. The same identity terminates the frontend listener and authenticates outbound connections to backends.
3. The peer’s verified SPIFFE ID is exposed to CEL policy as source.spiffeId.

If SPIFFE is enabled and the socket cannot be reached, the gateway fails to start rather than serving without an identity. That default is the right one and it is worth knowing before you deploy it.

The whole SPIFFE surface is three stanzas

Here is the config the demo runs. There is no certificate path in it, which is the point:

config:
  spiffe:
    endpoint: unix:///run/spire/sockets/agent.sock

binds:
- port: 3000
  listeners:
  - name: agents
    protocol: HTTPS
    tls:
      spiffe: {}
    routes:
    - policies:
        authorization:
          rules:
          - allow: 'source.spiffeId == "spiffe://example.org/ns/demo/sa/agent-alpha"'
        backendAuth:
          key: $UPSTREAM_API_KEY
        backendTLS:
          spiffe: {}
          subjectAltNames:
          - spiffe://example.org/ns/demo/sa/mock-llm
      backends:
      - host: mock-llm:8443

tls.spiffe terminates the listener with the SVID from the Workload API. Client certificates are mandatory in this mode and are verified against the trust domain bundle, so by the time the authorization rule reads source.spiffeId, it is a verified fact and not a client assertion.

backendTLS.spiffe handles the other leg. The gateway presents its own SVID to the upstream, and subjectAltNames pins which upstream identity it will accept. SVIDs carry a spiffe:// URI SAN and no DNS SAN, so ordinary hostname verification does not apply and this pin is how you narrow it.

backendAuth.key attaches the provider credential outbound. That is the part the agent never sees.

Identity is not authorization

The demo runs two agents that differ in exactly one respect: their SPIFFE ID. Both are attested by the same SPIRE agent, both hold valid SVIDs from example.org, both complete the TLS handshake with the gateway.

agent-alpha is in the CEL rule:

agent: my SPIFFE ID is spiffe://example.org/ns/demo/sa/agent-alpha
agent: provider credentials I hold: {'env': 'none', 'key_files': 'none'}
agent: HTTP 200
agent: Upstream saw client SPIFFE ID spiffe://example.org/ns/demo/sa/agentgateway and a valid provider credential.

agent-beta is not:

agent: my SPIFFE ID is spiffe://example.org/ns/demo/sa/agent-beta
agent: provider credentials I hold: {'env': 'none', 'key_files': 'none'}
agent: HTTP 403 authorization failed

Two things in the first output are worth reading carefully. The agent reports holding no provider credential, and it checks honestly: it walks its own environment for anything shaped like an API key and its own filesystem for anything shaped like key material, and finds neither. Yet it gets a 200 back. The upstream refuses to answer without the provider credential, so if a completion came back, the gateway attached one.

The second is that the upstream reports seeing sa/agentgateway, not sa/agent-alpha. The gateway authenticated to the model with its own identity over its own mTLS connection. The agent’s identity terminated at the gateway, which is what you want: the blast radius of a compromised agent is a 403, not a set of upstream credentials.

The demo’s whoami route makes the verification explicit. It is a directResponse whose body is built from a CEL expression:

directResponse:
  status: 200
  bodyExpression: '"verified client SPIFFE ID: " + source.spiffeId + "\n"'

client says:  spiffe://example.org/ns/demo/sa/agent-alpha
gateway says: verified client SPIFFE ID: spiffe://example.org/ns/demo/sa/agent-alpha

The client cannot influence the second line. There is no header to set.

Watching the certificate rotate underneath a running process

The demo issues five-minute SVIDs. SPIRE renews at roughly half the lifetime, and the SPIFFE client library swaps the certificate in place without the workload restarting, reconnecting, or asking:

[   0s] serial=ad928b7fcb3992f3 expires=15:58:34Z
[  30s] serial=ad928b7fcb3992f3 expires=15:58:34Z
[  60s] serial=38e419281166908e expires=16:01:02Z  <-- rotated

This is the operational argument for SPIFFE, separate from the security one. Short credential lifetimes are usually a tradeoff against operational pain, because something has to redistribute the new secret. Here nothing does. The five minutes is a number in a config file that could be one minute, and no runbook changes.

Three things that cost me time

unknown field 'spiffe'. The config surface is new in v1.5.0, and an older binary rejects it with a message that lists every field it does know:

Error: config.spiffe: unknown field `spiffe`, expected one of `enableIpv6`, `dns`, `localXdsPath`, ...

I hit this because the agentgateway binary on my machine was v1.0.1 while the demo runs the v1.5.0 image. Validate against the version you will actually run:

docker run --rm -v "$PWD/config:/config:ro" \
  cr.agentgateway.dev/agentgateway:v1.5.0 --validate-only -f /config/agentgateway.yaml

Environment substitution is $VAR, not %VAR%. I wrote key: "%UPSTREAM_API_KEY%". The config validated cleanly, the gateway started, mTLS worked in both directions, the CEL policy passed, and then the upstream returned:

agent: HTTP 401 {"error": {"message": "missing or invalid provider credential", "type": "invalid_request_error"}}

The gateway had faithfully sent the literal string %UPSTREAM_API_KEY% as the credential. A wrong-syntax placeholder is not a config error, it is a valid string, so this surfaces as an authentication failure several hops away from its cause. Worth noting that --validate-only resolves the variable too, so validate with it set or you get error looking key 'UPSTREAM_API_KEY' up: environment variable not found.

The SPIRE server needs a writable data directory. The SPIRE images are distroless and run as uid 1000, and only /opt/spire and /opt/spire/bin exist inside them. Mount a named volume at /opt/spire/data/server and Docker creates that path root-owned, so the server dies immediately:

level=error msg="Fatal run error" error="datastore-sql: datastore-sql: unable to open database file: no such file or directory"

The message points at the database file, but the file is missing because the directory it would live in is not writable. This demo puts SPIRE’s state under /tmp, since the bootstrap script recreates the trust domain from nothing on every run.

Running it

Prerequisites are Docker with Compose v2. There is no cluster, no cloud account, and no provider key to supply.

git clone https://github.com/themsquared/agent-identity-spiffe.git
cd agent-identity-spiffe
./scripts/bootstrap.sh
./scripts/demo.sh

bootstrap.sh starts the SPIRE server, exports its trust bundle, mints a one-time join token, attests the node with it, registers one entry per workload keyed on a docker label, and brings up the gateway, the upstream, and both agents. About a minute on a warm image cache.

The claims are asserted rather than narrated:

./scripts/verify.sh

verifying...
  ok    no certificate or key files in the repo
  ok    gateway config contains no cert or key path
  ok    agent-alpha gets HTTP 200
  ok    agent-alpha holds no provider credential
  ok    upstream authenticated the gateway's SVID
  ok    agent-beta gets HTTP 403
  ok    gateway echoes the verified SPIFFE ID

7 passed, 0 failed

What changes on Kubernetes

The shape transfers and the moving parts get smaller. SPIRE runs as a DaemonSet, the Workload API socket arrives through a CSI driver instead of a compose volume, and the workload attestor selects on namespace and service account rather than a docker label. The gateway config changes only in the socket path.

I used the ns/<namespace>/sa/<serviceaccount> ID shape in this demo deliberately, because it is what the Kubernetes workload attestor produces. The CEL rules move across without editing, and if you are already running Istio you have most of this infrastructure deployed.

One limit to plan around: v1.5.0 accepts only SVIDs chaining to its own trust domain bundle. SPIFFE federation across trust domains is not supported, so if your agents and your models live in different trust domains, that boundary needs a different answer today.

What I would build next

The obvious extension is dropping the CEL allowlist in favor of policy that reads the SPIFFE ID path structure, so ns/research/sa/* maps to a set of models without naming every agent. The interesting one is tying the SPIFFE ID to per-identity budgets, which would compose this with the per-key spend controls from v1.5.0: an identity that cannot be forged is a much better key to bill against than an API key that can be shared.

The demo, with all four claims and the scripts that check them, is at themsquared/agent-identity-spiffe.