# Your Agent's LLM Key Survives Every Kubernetes Secret Control > RBAC, restricted Pod Security, and automountServiceAccountToken all on, and one file read still hands over an agent's LLM API key. Tested on a live cluster. - Canonical URL: https://webofmike.com/agent-llm-key-kubernetes-controls/ - Author: Mike Moore (https://webofmike.com/about/) - Published: 2026-09-13 - Last modified: 2026-09-13 - Tags: Kubernetes, Security, AI Gateways, Platform Engineering, Code - Cite as: Mike Moore, "Your Agent's LLM Key Survives Every Kubernetes Secret Control", Web of Mike (webofmike.com), 2026-09-13. https://webofmike.com/agent-llm-key-kubernetes-controls/ I gave an agent pod an LLM API key the normal way, as a Secret projected into its environment. Then I turned on every Kubernetes control that sounds like it protects secrets: the `restricted` Pod Security Standard enforcing at admission, RBAC granting the agent's service account nothing at all on Secret objects, `automountServiceAccountToken: false`, a read-only root filesystem, all capabilities dropped. Every one of those is real and verifiable in the demo. Then one file read inside the container handed over the key, and the stolen value authenticated to the provider from a different pod. The code is at [themsquared/k8s-agent-secrets-hardening](https://github.com/themsquared/k8s-agent-secrets-hardening). It runs on kind, takes about 25 seconds to prove after the cluster is up, and ships 17 assertions so you can check the claims rather than take my word for them. The finding is narrow and worth stating precisely. **Kubernetes admission and RBAC controls govern access to Secret objects. They are not controls on a credential a pod has already been given.** Those are different things, the checklists blur them, and agents make the gap expensive because an agent is a workload whose whole job is to take instructions from text it did not write. I wrote [a control map for the July 2026 agent intrusion](/rogue-agent-kubernetes-controls/) that put this on a checklist item and moved on. This post is me actually running the checklist item to see what it buys. ## What the demo turns on Three namespaces. `provider` runs a stand-in LLM that checks its credential and returns 401 for anything else, so every 200 below is a real authentication result. `agents` runs the workloads under `restricted` enforcement. `gateway` runs agentgateway v1.5.0. The agent pod in stage A is not a strawman. It is what a careful team writes: ```yaml spec: serviceAccountName: agent-a automountServiceAccountToken: false securityContext: runAsNonRoot: true runAsUser: 65532 seccompProfile: {type: RuntimeDefault} containers: - name: agent securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsNonRoot: true capabilities: {drop: ["ALL"]} envFrom: - secretRef: {name: llm-provider-key} ``` Its Role grants `get` and `list` on ConfigMaps and nothing else. No verbs on secrets, anywhere. The enforcement is not decorative. A privileged pod in that namespace is rejected before it exists: ``` $ kubectl -n agents run privileged-probe --image=python:3.12-slim --privileged Error from server (Forbidden): pods "privileged-probe" is forbidden: violates PodSecurity "restricted:latest": privileged (container "p" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "p" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "p" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "p" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "p" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") ``` And the RBAC denial is genuine: ``` $ kubectl auth can-i get secrets --as=system:serviceaccount:agents:agent-a -n agents no $ ls /var/run/secrets/kubernetes.io/serviceaccount # inside agent-a ls: cannot access '/var/run/secrets/kubernetes.io/serviceaccount': No such file or directory ``` The agent cannot read Secret objects and cannot reach the API server at all. Both true, both verified in the demo. ## The credential is still readable Inside the agent container, with no code execution and no privilege escalation, just a file read: ``` $ python3 /src/steal.py read 1 {"pid": "1", "credentials_found": {"GPG_KEY": "7169605F62C751356D054A26A821E680E5FA6305", "LLM_API_KEY": "sk-provider-2f9c41e8"}} ``` Then the replay, from `bystander`, a third pod in the same namespace that holds no secrets and has no permissions: ``` $ python3 /src/steal.py replay http://mock-llm.provider.svc.cluster.local:8088 sk-provider-2f9c41e8 {"replay_status": 200, "provider_said": "{\"id\": \"chatcmpl-hardening\", ...}"} ``` 200 from a pod that was never given anything. The demo also replays a wrong key and gets a 401, which is the assertion that makes the 200 mean something. ## Why RBAC never sees the read This is the part that is obvious once stated and easy to carry the wrong intuition about for years. RBAC authorizes requests to the API server. A projected Secret does not make one. The kubelet reads the Secret object using its own credentials, at pod start, and writes the value into the container's environment. By the time the agent process exists, the transaction is over and the API server is not in the path. There is no request left to authorize. So `get secrets: no` is a true statement about a request the workload was never going to make. The control is aimed at a path the credential does not travel. `automountServiceAccountToken: false` has the same shape. It removes the pod's ability to talk to the API server, which is a genuinely good control and it is why the token directory does not exist above. It does not touch `LLM_API_KEY`, because that credential did not arrive through the API server either. ## What each control actually buys | Control | Blocks reading the Secret object | Blocks reading the credential the pod holds | |---|---|---| | RBAC deny on `get secrets` | yes | no | | `automountServiceAccountToken: false` | yes, no API access at all | no | | Pod Security Standard `restricted` | not its job | no | | `readOnlyRootFilesystem: true` | not its job | no | | Credential held by the gateway | n/a | yes, there is nothing to read | | NetworkPolicy on agent egress | n/a | no, but a stolen key cannot be spent | The left column is not useless. Every row of it is worth turning on and the intrusion writeups are full of stages those controls would have broken. The point is that the right column is a different question, and no amount of the left column answers it. ## readOnlyRootFilesystem and /proc Worth its own paragraph because it is the one I see cited most often as credential protection. `/proc` is not the root filesystem. It is a separate procfs mount, and `readOnlyRootFilesystem` makes the container's root mount read-only, which is a write control. `/proc/1/environ` stays readable regardless. Nothing about mounting the root filesystem read-only makes a process's own environment private to it. ## Stage B: the agent holds no credential One change. `agent-b` runs the same `app/agent.py` as `agent-a`, with the same security context, in the same namespace under the same enforcement. Its `envFrom` block is deleted and `TARGET_URL` points at agentgateway instead of the provider. The gateway config is short. It holds the key and attaches it outbound: ```yaml llm: gateways: default/llm providers: - name: provider provider: custom: formats: - type: completions params: baseUrl: http://mock-llm.provider.svc.cluster.local:8088 defaults: auth: key: $PROVIDER_API_KEY models: - name: demo-model provider: reference: provider ``` The agent still gets its completion: ``` $ python3 /src/agent.py "hello from stage B" {"target": "http://agentgateway.gateway.svc.cluster.local:3300", "agent_holds_key": "no", "status": 200, "answer": "billed to the provider key in this request"} ``` And the same read primitive that dumped the key in stage A comes back with nothing spendable: ``` $ python3 /src/steal.py read 1 {"pid": "1", "credentials_found": {"GPG_KEY": "7169605F62C751356D054A26A821E680E5FA6305"}} ``` I left that `GPG_KEY` in rather than filtering it out of the output, because a demo whose punchline is an empty dict invites the suspicion that the dict was emptied by the grep. It is the public fingerprint `python:3.12-slim` bakes in to verify CPython tarballs. The demo replays it at the provider and asserts a 401, so the claim is not that the read found nothing, it is that the read found nothing that buys anything. This is the same argument as [an agent that holds no LLM credential at all](/secretless-ai-agents/), moved onto Kubernetes and pointed at the specific controls people expect to substitute for it. ## The NetworkPolicy that kind silently ignores Stage B also restricts `agent-b` egress to the gateway and DNS. That control does bite, on a different axis: it means a credential obtained some other way still cannot reach the provider. ``` $ TARGET_URL=http://mock-llm.provider.svc.cluster.local:8088 python3 /src/agent.py "bypass" {"target": "http://mock-llm.provider.svc.cluster.local:8088", "agent_holds_key": "no", "status": 0, "answer": "connection failed: timed out"} ``` Getting that to be true took a change to the cluster itself. kind's default CNI does not enforce NetworkPolicy. It accepts the objects, stores them, returns them from `kubectl get`, and passes the traffic anyway. So `kind/cluster.yaml` sets `disableDefaultCNI: true` and `up.sh` installs Calico. That failure mode deserves more attention than the demo gives it. A NetworkPolicy that is accepted by the API server and silently does nothing looks exactly like a NetworkPolicy that works, from every angle except a packet. If you have egress policies in a cluster and have never watched one actually refuse a connection, that is worth half an hour this week. ## Gotchas **`kind load docker-image` fails on a multi-arch tag.** The first version of `up.sh` preloaded images to avoid registry pulls: ``` ERROR: failed to load image: command "docker exec --privileged -i agent-secrets-control-plane ctr --namespace=k8s.io images import --all-platforms --digests --snapshotter=overlayfs -" failed with error: exit status 1 Command Output: ctr: content digest sha256:c79067e2b6a0c38bd90c0c2d7de937eb0641f5d6e0329c61d982f3cdf41867df: not found ``` A local pull of a multi-arch tag on Apple silicon gives you one platform's layers plus a manifest list that references the others. kind imports with `--all-platforms`, so `ctr` looks for content that was never on the machine. Re-pulling does not help. The fix in the repo is to delete the step and let containerd pull, which is the path a reader's cluster takes anyway. **The `restricted` profile will reject your demo pods too.** Labelling the namespace and then writing an ordinary pod spec gets you an admission failure, not the comparison you wanted. Every pod in `agents` had to be written compliant first, which is the right way round: the interesting claim is that a fully compliant pod still leaks, and that claim requires the pod to actually be compliant. ## Run it ```bash git clone https://github.com/themsquared/k8s-agent-secrets-hardening cd k8s-agent-secrets-hardening ./scripts/up.sh # kind cluster, Calico, all manifests ./scripts/demo.sh # four acts, printed as they run ./scripts/verify.sh # 17 assertions, exit 0 if the post is accurate ./scripts/down.sh ``` Docker, kind and kubectl. No cloud account, no provider key. The cluster is named `agent-secrets` and nothing else on the machine is touched. ## What this does and does not settle Settled: the four controls in the left column of that table do not protect a credential the pod already holds, and the reason is structural rather than a misconfiguration. Also settled: moving the credential to a gateway removes it from the pod, and that is checkable rather than asserted. Not settled: the gateway in this demo has no inbound authentication, so any pod that can reach port 3300 can spend the key. The NetworkPolicy narrows who that is, and it is a network control standing in for an identity control. The identity half needs a token or an SVID, which is [the secretless demo](/secretless-ai-agents/) and [the SPIFFE one](/spiffe-identity-for-ai-agents/). A production version is both: the credential lives at the gateway, and the gateway knows who is asking. Also not settled, and the harder problem: stage B moves the credential out of the agent's reach but the agent can still ask the gateway to spend it. That is a budget and authorization question, not a secrets question, and it is [where per-key limits and tool allowlists come in](/agents-md-not-a-security-control/). The repo is at [themsquared/k8s-agent-secrets-hardening](https://github.com/themsquared/k8s-agent-secrets-hardening). If you turn one thing on this week, turn on the thing in the right column.