I built a demo where an agent calls an LLM and has no LLM credential anywhere in its process. No API key in the environment, no key file on disk, nothing in /app worth stealing. It asks a local issuer for an identity token that lives 60 seconds, sends that to agentgateway, and the gateway decides what the agent may do and attaches the real provider credential on the way out. Everything runs on docker compose with no cloud account and no provider key. The code is at themsquared/secretless-agents.
The reason to build it now is the LiteLLM supply chain compromise in March.
What the LiteLLM compromise actually took
On March 24, 2026, a LiteLLM maintainer’s GitHub account was taken over and the release pipeline was used to publish versions 1.82.7 and 1.82.8 to PyPI with a credential stealer inside. The packages were live for roughly 40 minutes and were downloaded tens of thousands of times before PyPI quarantined them. Sonatype’s analysis and InfoQ’s writeup have the full timeline.
Two details are the ones that matter for how you build.
The payload shipped as a .pth file in the wheel. CPython executes .pth files in site-packages at interpreter startup, so the code ran without anyone writing import litellm. Installing the package was enough. Every piece of advice that starts with “audit your dependencies before you import them” was already too late.
And what it harvested was environment variables, .env files, cloud credentials, Kubernetes configs, and SSH keys. That list is not a coincidence. It is a precise description of where an LLM provider API key lives in a normal agent deployment.
Why a better secret store does not fix this
The instinct after a breach like this is to move the key somewhere safer. Vault instead of an env var. A CSI driver instead of a mounted file. Short-lived cloud credentials fetched at boot.
All of that improves how the secret gets to the process. None of it changes the fact that once the process has it, an attacker running inside that process has it too. A secret manager protects a credential at rest and in transit. It does nothing about the moment the application holds the plaintext, which is the moment the .pth payload was waiting for.
The sk- key is also uniquely bad to lose. It does not expire. It is not scoped to a model, a team, or a tenant. It does not distinguish your agent from anyone holding the same string. And its theft is invisible until either the provider’s anomaly detection or your finance team notices.
So the question is not where to keep the key. It is whether that process needs a key at all.
Two credential boundaries
It does not. The agent needs to prove who it is, and something else needs to hold the provider credential. That splits into two boundaries.
Inbound. The agent authenticates to the gateway with a short-lived identity token from a workload identity system. In the demo that is a 60-second RS256 JWT from a small local issuer. In a real cluster it is SPIFFE, an OIDC provider, or the cloud’s workload identity service. The point of the token is that it says who the agent is and nothing else. It is not a bearer credential for any provider.
Outbound. The gateway holds the provider credential and attaches it. In the demo one model gets a long-lived key, and the other has agentgateway mint a fresh signed JWT per request, so nothing long-lived exists on that connection either.
agent issuer agentgateway mock-llm
(no secrets) (signs identity) (holds credentials) (checks them)
| | | |
|-- POST /token ->| | |
|<- 60s JWT ------| | |
| | |
|-- Bearer <60s JWT> ------------------->| |
| jwtAuth: strict, JWKS from issuer |
| per-model CEL on jwt.team |
| | |
| |-- provider key ---->| /static
| |-- minted JWT ------>| /signed
|<- 200 -------------------------------- | |
The mock upstream in this demo returns 401 when the credential is missing or wrong. That detail is what makes the whole thing an assertion rather than a claim: if the agent gets a response at all, the gateway must have attached something the agent never held.
Here is what the agent container actually has:
docker compose exec agent sh -c 'env | sort | grep -v ^PATH='
GATEWAY_URL=http://agentgateway:3300
GPG_KEY=7169605F62C751356D054A26A821E680E5FA6305
HOME=/root
HOSTNAME=4638977af137
ISSUER_URL=http://issuer:8099
LANG=C.UTF-8
PWD=/app
PYTHON_SHA256=5c8462af5790baf43a321a1559dbe0db06d1be4300fb85fb53c40060668e548a
PYTHON_VERSION=3.12.14
UPSTREAM_URL=http://mock-llm:8088
Three URLs and some Python build metadata.
How agentgateway verifies the agent’s identity
The inbound half is one policy block on the LLM listener. mode: strict means a request without a valid token from this issuer never reaches a model:
llm:
policies:
jwtAuth:
mode: strict
issuer: "https://issuer.secretless.local"
audiences: ["agentgateway"]
jwks:
file: /keys/jwks.json
The mode field is the load-bearing part. The default is optional, which validates a JWT when one is present and lets the request through when it is not. That default is reasonable for a gateway that fronts a mix of authenticated and public routes, and it is exactly wrong for a gateway holding a provider API key. Set it to strict or you have built an open proxy that spends your money.
With strict, an anonymous call fails before it costs anything:
docker compose exec agent python3 /app/agent.py --no-token
HTTP 401
authentication failure: no bearer token found
Identity is not authorization
A valid token proves the agent is who it says. It does not say what the agent may do. Those are separate decisions and agentgateway keeps them separate.
The gateway-level rule is deliberately weak. It requires a team claim and stops there:
authorization:
rules:
- allow: 'has(jwt.team)'
What each team may actually call is decided per model:
models:
- name: secretless-static
authorization:
rules:
- allow: 'jwt.team in ["research", "platform"]'
- name: secretless-signed
authorization:
rules:
- allow: 'jwt.team == "platform"'
A research agent presenting a perfectly valid, perfectly fresh token, asking for the model its team is not cleared for:
docker compose exec agent python3 /app/agent.py --agent research --model secretless-signed
HTTP 403
{
"error": {
"message": "Model authorization denied",
"type": "invalid_request_error",
"code": "model_authorization_denied"
}
}
That is the property worth having. A stolen token is bounded by what its team was allowed to do, not by what the gateway is capable of.
Why the model rules are per model
My first version tried to do this in one place, with a gateway-level rule matching on the requested model:
authorization:
rules:
- allow: 'jwt.team == "research" && llm.requestModel == "secretless-static"'
Every request got a 403, including ones that should have passed. The jwt half was fine, which I confirmed by cutting the expression down to jwt.team == "research" and watching it return 200. The llm object is not populated during the gateway-level authorization phase, so llm.requestModel does not evaluate to the model name there. Moving the model check onto the model entry, where that context exists, fixes it. The per-model form is clearer anyway.
How the provider credential gets attached
Two providers, two upstream auth styles.
The first is the familiar one, with the key moved off the agent and onto the gateway:
- name: static-key
params:
baseUrl: http://mock-llm:8088/static
defaults:
auth:
key: $UPSTREAM_API_KEY
This is already most of the win. The key exists in one process that does not execute agent code, does not install packages at runtime, and does not import a model provider SDK.
The second is the one worth building toward. jwtSign has agentgateway sign a fresh JWT with its own private key on every single request:
- name: signed-jwt
params:
baseUrl: http://mock-llm:8088/signed
defaults:
auth:
jwtSign:
signingKey:
file: /keys/gateway-sign.key
alg: RS256
ttl: 60s
claims:
iss: agentgateway
aud: mock-llm
sub: agentgateway/llm-egress
Now there is no long-lived bearer token on that connection to capture at all. The upstream verifies the signature and reports how much life the token it received has left:
docker compose exec agent python3 /app/agent.py --agent platform --model secretless-signed
HTTP 200
authenticated with a JWT agentgateway minted for this request, sub=agentgateway/llm-egress expires_in=60s
This only works against upstreams that verify keypair JWTs rather than a static key. agentgateway’s config schema gives the Snowflake SQL API as its example; the major model providers still authenticate with a static API key, so for those you are on the key form above. Where you do control the upstream, jwtSign is worth wiring up, because it is the version of this pattern with no standing credential anywhere on the path.
The breach test
The honest way to evaluate any of this is to assume the agent is fully compromised, take everything it has, and ask what that is worth.
What it has is a 60-second identity token. Sent straight at the provider, bypassing the gateway:
docker compose exec agent python3 /app/agent.py --agent platform --direct
HTTP 401
{
"error": {
"message": "upstream rejected the request: wrong or missing provider key",
"type": "invalid_request_error"
}
}
An identity assertion is not a provider credential. A minute later it is not even a valid identity assertion.
The reverse holds too. If the provider key did leak from somewhere else, it is not a gateway credential:
curl -s -o /dev/null -w '%{http_code}\n' -X POST http://localhost:3300/v1/chat/completions \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-upstream-do-not-leak' \
-d '{"model":"secretless-static","messages":[{"role":"user","content":"hi"}]}'
401
Two credentials, two boundaries, and neither one is a skeleton key for the other.
Gotchas
Four things cost me time. All four are in the repo’s README as well.
An expired token is accepted for 60 more seconds. agentgateway allows 60 seconds of clock skew on exp. A token that expired 30 seconds ago validates; one that expired 61 seconds ago does not. I found this because my “expired token is rejected” assertion failed with an HTTP 200:
docker compose exec agent python3 /app/agent.py --ttl -30 # HTTP 200
docker compose exec agent python3 /app/agent.py --ttl -120 # HTTP 401
authentication failure: the token is invalid or malformed: Error(ExpiredSignature)
This is normal JWT library behavior and it is not a bug, but the number matters when you are choosing a token lifetime. A 60-second token has a real window closer to 120 seconds.
failed to load JWKS: read resource file /keys/jwks.json. The gateway resolves the JWKS when it loads the config, not lazily on the first request. Keys have to exist before it starts. In the demo that is why key generation is a one-shot compose service the others wait on with condition: service_completed_successfully rather than a line in a shell script. The field also takes {url: ...} for a remote JWKS endpoint, which is what a real deployment uses; the demo mounts a file so docker compose up is deterministic and does not race the issuer coming up.
error looking key 'UPSTREAM_API_KEY' up: environment variable not found. agentgateway expands $VAR in the config at load time and fails closed when the variable is unset. That is the behavior you want, and it means --validate-only needs the same environment as the real run:
docker run --rm -e UPSTREAM_API_KEY=dummy \
-v "$PWD/config:/config:ro" -v "$PWD/keys:/keys:ro" \
cr.agentgateway.dev/agentgateway:v1.5.0 -f /config/agentgateway.yaml --validate-only
Configuration is valid!
upstream call failed: Connect: Connection refused (os error 111). The gateway resolved the upstream address once and held it. Recreating only the upstream container gives it a new IP, and every request 503s until the gateway restarts. docker compose restart agentgateway clears it. A Kubernetes Service address hides this; raw compose DNS surfaces it immediately, and it is easy to misread as a broken config.
Run it
Requirements are Docker with Compose v2. No provider account and no API key.
git clone https://github.com/themsquared/secretless-agents.git
cd secretless-agents
docker compose up -d --build
./scripts/verify.sh
verify.sh is 14 assertions, not narration. It checks that the agent’s environment and filesystem hold no provider key, that both upstream auth styles work, that a missing token and an expired token are both rejected, that the skew window is exactly what I said it is, that the provider key is not a gateway credential, that a valid token is still denied the model its team may not use, and that going around the gateway fails at the provider.
14 passed, 0 failed
For a narrated walk through the same facts, ./scripts/demo.sh.
What this does and does not settle
The demo is small on purpose. Two CEL rules on one claim is the least it takes to show that identity and authorization are different decisions. The upstream is a mock, so nothing here is a benchmark. And the token lifetime, the issuer, and the claim shape are all things you would replace with your existing workload identity.
What it does settle is the shape. The provider credential belongs in a process that does not run agent code, and the agent gets an identity instead. When the next package in the agent’s dependency tree gets taken over, and there will be a next one, what the attacker finds in that process is a token that expires in a minute and cannot buy anything.
The code is at themsquared/secretless-agents. The natural next piece is replacing the toy issuer with SPIFFE, so agent identity comes from the workload’s own attestation rather than a service handing out tokens to whoever asks. If you want the cost side of the same gateway, I wrote up budgets and virtual keys earlier.