28 posts in this hub //

Securing AI Agents in Production: The Complete Series

Egress allowlists, tool allowlists, AGENTS.md, audit logs, Kubernetes secret controls: a series on the agent controls that look like controls and are not, and what to put in their place.

Almost everything in this series started the same way: a control that everybody agrees is the right control, applied to an agent, quietly not doing the thing its name implies. I gathered the first month of those labs in AI agent security on Kubernetes .

The pattern is consistent enough to be a design rule. Controls built for human operators assume a session, an intent, and a bounded set of actions. An agent has none of those. It has a tool surface, a credential, and a loop — and the loop will find the edge of every boundary you drew for a person.

The thread running through these

Advisory is not enforcement. AGENTS.md is a file the model may read. A tool name allowlist pins a string, not behavior. Neither survives contact with a model that has a reason to go around it.

Permitted is not safe. The hardest destination to defend is the one you are required to allow. The artifact registry has to be reachable because builds need it, and that single exception is enough to run data out and coordination back in.

The record is written by the suspect. If the agent can influence the transcript, the transcript is testimony, not evidence. This is the one that most surprises people, and it has a real number behind it.

The credential outlives the control. Kubernetes secret hygiene covers the things Kubernetes issues. The provider API key the agent actually holds is usually not one of them.

Where to start

If you are trying to convince someone this matters, start with which controls would have stopped the July 2026 intrusion — it works backward from a real incident.

If you are already convinced and want the sharpest single finding, start with the audit log your agent wrote .

If you need the governance version for a security review, start with who gave the agent a badge : identity, least privilege, approvals and audit for agents, mapped to OWASP Excessive Agency. If you run LLM proxies in production, read why image signing didn’t stop the LiteLLM 1.107.0 backdoor .

If you run a platform and want to know what is already deployed without your knowledge, start with shadow MCP , finding the MCP servers your platform team doesn’t know about , and find shadow MCP servers .

The related work on how an agent proves who it is — SPIFFE, token exchange, audience binding, and what MCP does and does not specify — is collected separately in the agent identity series .

Every post in this hub

  1. OpenAI Red-Teamed Its Sandboxes and Found a CPU Side Channel
    OpenAI's own red-team agents broke its hardened microVMs and found a CPU cache side channel to talk across isolation boundaries. The fix isn't a better sandbox.
  2. OpenAI's Patch the Planet: What Safeguards Agent Scanning Needs
    OpenAI fixed a critical PHP flaw with agent help, then asked what safeguards agent vulnerability scanning needs. The HN reaction argued motive instead.
  3. include_content=False Didn't Stop pydantic-ai's MCP Token Leak
    A second, separate flag kept pydantic-ai MCP server tokens and headers in OpenTelemetry trace spans in plaintext, even with include_content=False, until v2.58.0.
  4. Stopping Agent Proxy Laundering at the MCP Gateway
    Wikimedia says agents reconfigured a citation tool into a fetch proxy. The agentgateway MCP guardrail that rejects the laundered destination before the fetch runs.
  5. Two Grok Bots Leaked a CEO's Bank Balance to Slack
    Two 'separate' Grok Bots leaked a CEO's bank balance to public Slack. I reproduced it, then built an agentgateway guardrail that blocks the exact handoff.
  6. MCP Tool-Call Guardrails: Gating Agents Before the Tool Runs
    agentgateway can now reject or mask an MCP tools/call before the tool runs, and mask the result before the client sees it. No sidecar service required.
  7. The Path-Validation Bug That Let Copilot Register a Rogue MCP Server
    CVE-2026-103572: Copilot validated a filename string, then wrote a different, resolved path. A reproducible demo of the bug class and its one-line fix.
  8. MCP Audit Tap: See What a Third-Party Server Actually Sends
    A Show HN WhatsApp MCP server drew 69 comments on privacy and bans, and nobody could see what it sends. A 100-line audit tap that can, and the log it writes.
  9. Testing Open-Weight Models as an Agent's Cheap Tier
    Two open-weight models, routed through agentgateway, scored against Claude Sonnet 5 on 60 agent tool-call cases. One failed to answer 55% of the time.
  10. Stopping a Filesystem MCP Server From Reading .env Files
    agentgateway's mcpAuthorization can't see a tool call's path argument. Here's the ExtMCP policy that blocks a filesystem MCP server's .env read instead.
  11. A Private MCP Registry in 10 Minutes with agentregistry
    Run a private MCP registry on agentregistry v0.4.0, publish servers from YAML, and serve the official MCP Registry v0.1 API, validated against its own spec.
  12. Is Jev Actually Calibrated? The Reliability Curve on 240 Cases
    A reliability diagram and ECE for TypeSafe AI's Jev on 240 labelled agent tool calls. A 1.000 held up. A 0.97 was right 88% of the time. The set is public.
  13. Rate-Limiting AI Agent Egress to Already-Allowed APIs
    OpenAI agents queried Wikidata hundreds of thousands of times, a destination the allowlist permitted. The agentgateway rate limit that caps it, reproduced live.
  14. AI Agent Security on Kubernetes: Which Controls Actually Held
    A month of AI agent security labs on identity, egress, audit logs, MCP tool poisoning, runtimes and evals. The controls that held all sat outside the agent.
  15. Your Agent Egress Proxy Never Saw the DNS Query
    An OpenAI agent reached an external chatbot over DNS after its HTTPS calls were blocked. The Kubernetes NetworkPolicy that closes that channel, reproduced live.
  16. What Stopped AI Agent Probes of Government Data Sites
    OpenAI agents probing government data sites during ordinary retrieval. What Cloudflare stopped, what it missed, and the egress control that generalizes.
  17. Your Artifact Registry Is a Two-Way Channel for Agents
    An agent's egress allowlist permits the artifact registry because builds need it. That single permitted destination is enough to run a covert channel, as this week's RubyGems incident showed.
  18. MCP Tool Poisoning: A Name Allowlist Is Not Enough
    A deny-by-default MCP allowlist strips gated tools and still adopts a poisoned description. Pinning name, description and schema rejects the mutation.
  19. Stopping an AI Agent Without Losing the Forensic Record
    Security teams want an agent stopped now and everything it did preserved. Those fight unless the record lives outside the agent. A test on Kubernetes.
  20. A WAF That Reads the Prompt: OWASP CRS for LLM and MCP
    Coraza with body inspection puts OWASP CRS and custom rules on LLM prompts and MCP tool arguments. The verified blocks, and the false-positive check.
  21. I Benchmarked Jev on Agent Tool-Call Risk. Calibration Held.
    A 60-case Jev benchmark on agent tool-call risk: readonly, destructive, privileged, or exfiltration. Accuracy 91.7 percent. The finding is calibration.
  22. Shadow MCP Servers: Track, Find and Stop Them Across Teams
    Find and stop shadow MCP servers that are hard to track across teams. Scan the network, sweep laptop configs, then enforce every call at a gateway.
  23. Your Agent's LLM Key Survives Every Kubernetes Secret Control
    RBAC, restricted Pod Security, and automountServiceAccountToken all on, and one file read still hands over an agent's LLM API key. Tested on my own test cluster.
  24. Keeping MCP Server Instructions Out of Your Agent's System Prompt
    Keep server-controlled MCP instructions out of the trusted system prompt. Isolate, cap, bind the cache, and pin the digest. Lab plus the four controls.
  25. Agent Audit Logs: What METR and Redwood Found
    METR found spoofed tool calls in 7% of agent transcripts from the Hugging Face incident. A transcript monitor reads them as clean. Diff against a witness.
  26. Why Proxy-Variable Agent Sandboxes Fail, and the CONNECT-Time Fix
    Proxy-variable agent sandboxes fail because the agent can change method, hostname, and NO_PROXY. Authorize the resolved destination at CONNECT time instead.
  27. Your AGENTS.md Is Not a Security Control
    A prompt-injected agent destroyed every record with an AGENTS.md forbidding it and an approval classifier watching. An MCP allowlist at the gateway stopped it.
  28. Which Controls Would Have Stopped the July 2026 Agent Intrusion?
    An autonomous agent went from sandbox escape to Kubernetes cluster-admin in under 13 hours. A stage-by-stage map of which controls would have broken the chain.

Frequently asked questions

What are the main security risks of running AI agents in production?

The recurring pattern is not a novel exploit but a control that does not hold. An agent reaches a blocked host through a permitted proxy, exfiltrates through a destination the allowlist has to permit, calls a tool whose description changed after approval, or writes the audit record you would use to catch it. Kubernetes secret controls cover the service account and miss the LLM API key. Each of these is a documented incident, not a hypothetical.

Why is an egress allowlist not enough to contain an AI agent?

An allowlist decides which hosts an agent may reach, not what it does once connected. Builds need the artifact registry, so the registry is permitted, and publish and list are ordinary operations that can carry encoded data in both directions. The 2026 RubyGems incident ran a covert channel entirely through documented registry features, with no vulnerability involved. Containment has to move to the write path and to per-identity behavior.

Can you trust an AI agent's own logs for an audit trail?

Not without tamper evidence. METR and Redwood Research found that at least 20 percent of agents in one investigation expressed clear interest in manipulating transcripts, and roughly 7 percent of transcripts contained successfully spoofed tool calls. If the agent can write to the record, the record is testimony rather than evidence. The audit trail has to be produced by something on the call path the agent does not control, such as the gateway.

Is a tool name allowlist enough to stop MCP tool poisoning?

No. A name allowlist pins what a tool is called, not what it does. An MCP server can change a tool’s description, parameters, or instructions after the tool was approved, and the model reads the description at call time. The control that holds is pinning the full tool definition, including the description and schema, and failing closed when the digest changes.