This is the home for the agent work I publish here: runtimes, identity, gateways, MCP, and the substrate under them. If you want the map, start with the field guide. If you want how an agent proves who it is, start with Agent Identity.
Start here
- Agent Substrate: the field guide
- Agent Identity
- Agent Runtimes
- AI Gateways
- MCP
- Agent Security
- kubectl Is Lying to You
- kagent Agent Substrate
- What is an AI agent gateway?
- What is an MCP gateway?
Agent Substrate
- kubectl Is Lying to You (Monday's Talk)
On Monday I'm giving a talk on what a live visualizer taught me about Agent Substrate, and why kubectl get pods shows quiet workers while agents are busy. - Does Agent Substrate's Checkpoint Survive a Held File Lock?
I tried to break Agent Substrate's checkpoint with a held file lock. It survived: gVisor virtualizes flock() inside its own sentry, not the host kernel. - Substrate Scope: a Live Visualizer for Agent Substrate
Substrate Scope is an open source live visualizer for Agent Substrate: worker bays, restore queues, snapshot storage, telemetry, and per-agent activity. - kagent Agent Substrate: Thousands of Agents
kagent Agent Substrate is built to run thousands of AI agents on tens of pods. I stood it up on kind: snapshot restores, autoscaling, and every gotcha I hit.
Agent Identity
- MCP Cross App Access Explained: The ID-JAG Exchange on the Wire
MCP Enterprise-Managed Authorization (Cross App Access) traced request by request: the ID-JAG exchange, the JWT bearer grant, and agentgateway validating it. - agentgateway Token Exchange for AI Agents
How AI agents act for a user without holding provider tokens: agentgateway token exchange (RFC 8693, JWT bearer, XAA) in standalone and Kubernetes. - Per-Request Minted JWTs for Agent Backends with agentgateway
agentgateway v1.5 signs a fresh 15-second JWT on every request, so an AI agent reaches its backend holding no key. Claims, expiry, and a failed replay, tested. - Do AI Agents Need SPIFFE or OAuth?
SPIFFE proves which agent process is calling; OAuth scopes what it may do and for whom. When each is enough, and when production agents need both. - Building Workload Identity Federation for AI Agents on Kubernetes
How an AI agent pod trades a Kubernetes service account token for a resource-scoped credential, with no shared secret and no long-lived key in the pod. - agentgateway CEL Gotchas: Fail-Open + 403 Fix
agentgateway matchExpressions are OR'ed, so two rules fail open; llm.requestModel is empty at the traffic phase, so every call 403s. The working policy. - The agentgateway Token Service Now Runs Without a Cluster
agentgateway enterprise 2026.9.0 ships its token service as a standalone binary on a public bucket, so a gateway-mediated agent identity setup fits on a laptop. - kagent Audience-Bound Tokens (RFC 8707)
kagent v0.10.0 GA can bind an agent's exchanged token to one backend using RFC 8707 resource indicators. Two environment variables, both empty by default. - MCP Agent Identity Gap: The Agent Has No Spec Yet
The MCP agent identity gap: one shipped spec authenticates the employee, not the agent. DPoP and Workload Identity Federation are still open pull requests. - SPIFFE for AI Agents: End-to-End Identity Demo
SPIFFE agent identity end to end: the agent, agentgateway v1.5.0 and the model upstream all authenticate with SPIFFE SVIDs. No certificate file exists. - Your AI Agent Should Not Hold the LLM API Key
How an AI agent calls an LLM with no API key in its env, disk, or process: short-lived identity at agentgateway, provider key attached at the gateway.
AI Gateways and MCP
AI Gateways ยท MCP
- Pydantic AI's DNS-Rebind Bypass of Its Egress Allowlist
Pydantic AI's GHSA-5ppx-9jq6-6pmw let a DNS rebind beat its egress allowlist. A runnable demo shows why the fix belongs in a gateway, not app code. - Best AI Agent Gateway Criteria for 2026
Criteria that decide if a product is an AI agent gateway, and what to ask before a shortlist: license, protocols, modes and releases. - Grok Bot on agentgateway codeSearch: 96% Fewer Tokens on Real GitHub Work
I put Grok Bot's GitHub MCP connector behind agentgateway Enterprise in codeSearch mode. A real PR review workflow went from 89,038 tokens to 3,809. - What Is an MCP Gateway? agentgateway Examples
An MCP gateway sits between AI agents and MCP servers: one endpoint, sign-in, per-tool CEL rules and audit. How it differs from LLM and API gateways. - Portkey Alternative: agentgateway, Self-Hosted
agentgateway is an open source Portkey alternative you run yourself: virtual keys, per-key USD or token budgets, guardrails and CEL policy for MCP tools. - How to Find Shadow MCP Servers in Your Environment
How to find shadow MCP servers on the network, in Kubernetes, on laptops, and in vendor accounts, then check them against a registry you approved. - agentgateway Guardrails How-To
Apply AI gateway guardrails with agentgateway: regex PII mask/reject, webhooks, Bedrock, Model Armor. S+K. Streaming and fail-open notes. Solo.io disclosure. - agentgateway as an Egress Proxy
agentgateway v1.6.0 HTTP CONNECT egress proxy for AI agents: hostname allowlist, default deny. Graded standalone proof; K ambient honest. Solo.io disclosure. - MCP Progressive Tool Discovery: Native vs. Hand-Rolled
A 3-tool MCP entry point cuts tools/list bytes 94% vs a flat 50-tool catalog. agentgateway's Search mode does this, Enterprise-only; here's the OSS version. - agentgateway vs Agent Router v1.2: MCP prefixMode Defaults
Agent Router v1.2 added prefixMode: Never for MCP tool names. agentgateway ships Conditional, Always and Never, defaulting to Conditional. How they differ. - What Is Shadow MCP? A Plain Definition
What is shadow MCP? Any MCP server your AI tools can reach that security did not approve and cannot see, plus where it comes from and why it is a risk. - agentgateway Standalone: Guardrails, Budgets, Tool Governance
A three-act agentgateway Enterprise demo in one docker-compose file: a guardrail gotcha on tool-result scanning, toolMode: code, and MCP tool authorization. - What Is an AI Agent Gateway?
An AI agent gateway governs model, MCP, A2A and HTTP traffic. How it differs from API, LLM and MCP gateways, and what agentgateway covers in both modes. - Prisma AIRS Guardrails for agentgateway in 150 Lines of Python
Connect Palo Alto Prisma AIRS to agentgateway with a guardrail webhook adapter that blocks prompt injection, masks SSNs and card numbers, and fails closed. - Agent Platform Gateway Control Plane
A production path: SPIFFE as the spine, then secretless LLM keys, spend caps that return 429, MCP tool allowlists, and egress control on one hop. - agentgateway v1.6.0: Cost Tracking With No Catalog Config
agentgateway v1.6.0 ships a built-in LLM pricing catalog, so cost tracking needs zero config, plus per-key CEL rate limits, validated against a live gateway. - Stateless MCP on Kubernetes: No More Sticky Sessions
A 3-replica MCP server behind agentgateway, no session affinity, survives a rolling restart mid-session. What the MCP 2026-07-28 spec changed, and the proof. - How to Choose the Best AI Agent Gateway: 7 Tests
How to choose an AI agent gateway by the controls it enforces, and how agentgateway meets each one in standalone and Kubernetes modes, with PoC tests. - What Is an Agentic Mesh (vs a Service Mesh)?
A platform-engineer definition of agentic mesh vs a service mesh: Solo/CNCF vs Microsoft AgentMesh, plus the controls that make it operational. - agentgateway vs LiteLLM vs Portkey, Compared
agentgateway vs LiteLLM: one Rust data plane for LLM, MCP and A2A, standalone or on Kubernetes, with per-key budgets. Portkey covered, with sources. - Serve Markdown to Agents at the Gateway, Not the Origin
An agentgateway route serves markdown to agents on the Accept header while the HTML-only origin stays untouched, plus the two ways a naive version fails. - Conway's Law Is Coming for Your MCP Servers
Putting every MCP tool behind one federated endpoint rebuilds the ESB and makes agents worse. Why MCP federation should follow business domain boundaries. - Cap LLM Spend per API Key: agentgateway 1.5
agentgateway v1.5.0 caps LLM spend per API key and limits which models a key may call. A runnable demo of the 429, the price math, and the gotchas. - Multi-Tenant MCP Servers: Per-Customer Tools
Run MCP servers multi-tenant: agentgateway gives each customer its own tools, quotas, and chargeback from a JWT claim. The servers hold zero auth code. - LLM Cost Controls: Cap Spend per Team with Virtual Keys
How virtual keys track and cap LLM spend per app or agent, and why to cap spend at the gateway: past the USD or token budget, calls return HTTP 429.
kagent
- The kagent Harness: Declarative Agents, Memory and Compaction
kagent's built-in harness runs a Go engine that reads your AgentTemplate: why it's the default, how memory and compaction work, and what you get that Claude Code, Codex and BYO don't. - kagent BYO Harness: Running Factory Droid as an A2A Agent
What kagent's byo harness requires, and how kagent-droid runs Factory Droid behind it: a Go A2A front, one droid exec per turn, session continuity on /data, and read-only k8s-tools. - kagent Harnesses Compared: kagent vs Claude Code vs Codex vs BYO
The four kagent 1.0 harnesses side by side: what each runs, which model providers it takes, what it drops from an AgentTemplate, and how to pick one. - kagent Codex Harness: Run OpenAI Codex on Kubernetes
How to run OpenAI Codex as a kagent 1.0 harness: the four resources, the ModelConfig rules, MCP and env restrictions, and how to debug a Compatible=False agent. - What AI Agent Harness Is Kubernetes-Native? kagent
kagent is a Kubernetes-native AI agent harness. What a cloud-native harness is, and how kagent splits harness, template, instance and sandbox. - Regression Tests for kagent Agents with agentevals
kagent records agent sessions as ADK events. agentevals scores chat-message trajectories. I built the bridge, and a suite that catches an ungrounded agent. - kagent Claude Code Harness on Kubernetes (Updated for v1alpha3)
Run Claude Code as a kagent harness: current Harness + AgentTemplate + Agent YAML, the MCP all-tools gotcha, plus the v1.0.0-alpha1 history (AgentInstance, checkpoints, golden snapshots).
Agent security
- Containing an Agent Backdoor Planted by Pretraining Poisoning
A new paper backdoors a CLI agent by poisoning 0.1% of its pretraining corpus. Here's why the fix isn't detecting the poison, it's containing what any agent can do. - Confidence-Routed Agents: Cheap First, Frontier on Escalation
A router calls Jev for agent tool-call risk, escalating to Claude via agentgateway only on low confidence. Escalating everything cost more and scored worse. - agentgateway v1.7.0: How Agent Session Detection Works
agentgateway v1.7.0 detects a coding agent's session from headers like x-claude-code-session-id, with an 8-character minimum that fails silently, no error. - Why Cloudflare's GLM-5.3 KV Cache Leaked Across Tenants
Cloudflare's postmortem: a KV cache pool size mismatch plus an SGLang admission bug let 155 Workers AI responses blend two different tenants' data together. - Solo.io's agentshield Watches Your Agent Perimeter for Drift
Solo.io's agentshield builds an attack-path graph across Kubernetes, Istio, Cilium, AWS, and agentgateway, trusting a model only when it cites graph evidence. - AI Agent Containment: Why a Sandbox Alone Isn't Enough
A sandboxed AI agent still needs a perimeter it cannot route around: default-deny egress, no ambient credentials, an audit log, one agentgateway config. - OpenAI Red-Teamed Its Sandboxes and Found a CPU Side Channel
OpenAI's own red-team agents broke its hardened microVMs and found a CPU cache side channel to talk across isolation boundaries. The fix isn't a better sandbox. - OpenAI's Patch the Planet: What Safeguards Agent Scanning Needs
OpenAI fixed a critical PHP flaw with agent help, then asked what safeguards agent vulnerability scanning needs. The HN reaction argued motive instead. - include_content=False Didn't Stop pydantic-ai's MCP Token Leak
A second, separate flag kept pydantic-ai MCP server tokens and headers in OpenTelemetry trace spans in plaintext, even with include_content=False, until v2.58.0. - Stopping Agent Proxy Laundering at the MCP Gateway
Wikimedia says agents reconfigured a citation tool into a fetch proxy. The agentgateway MCP guardrail that rejects the laundered destination before the fetch runs. - Two Grok Bots Leaked a CEO's Bank Balance to Slack
Two 'separate' Grok Bots leaked a CEO's bank balance to public Slack. I reproduced it, then built an agentgateway guardrail that blocks the exact handoff. - MCP Tool-Call Guardrails: Gating Agents Before the Tool Runs
agentgateway can now reject or mask an MCP tools/call before the tool runs, and mask the result before the client sees it. No sidecar service required. - The Path-Validation Bug That Let Copilot Register a Rogue MCP Server
CVE-2026-103572: Copilot validated a filename string, then wrote a different, resolved path. A reproducible demo of the bug class and its one-line fix. - MCP Audit Tap: See What a Third-Party Server Actually Sends
A Show HN WhatsApp MCP server drew 69 comments on privacy and bans, and nobody could see what it sends. A 100-line audit tap that can, and the log it writes. - Testing Open-Weight Models as an Agent's Cheap Tier
Two open-weight models, routed through agentgateway, scored against Claude Sonnet 5 on 60 agent tool-call cases. One failed to answer 55% of the time. - Stopping a Filesystem MCP Server From Reading .env Files
agentgateway's mcpAuthorization can't see a tool call's path argument. Here's the ExtMCP policy that blocks a filesystem MCP server's .env read instead. - A Private MCP Registry in 10 Minutes with agentregistry
Run a private MCP registry on agentregistry v0.4.0, publish servers from YAML, and serve the official MCP Registry v0.1 API, validated against its own spec. - Is Jev Actually Calibrated? The Reliability Curve on 240 Cases
A reliability diagram and ECE for TypeSafe AI's Jev on 240 labelled agent tool calls. A 1.000 held up. A 0.97 was right 88% of the time. The set is public. - Rate-Limiting AI Agent Egress to Already-Allowed APIs
OpenAI agents queried Wikidata hundreds of thousands of times, a destination the allowlist permitted. The agentgateway rate limit that caps it, reproduced live. - AI Agent Security on Kubernetes: Which Controls Actually Held
A month of AI agent security labs on identity, egress, audit logs, MCP tool poisoning, runtimes and evals. The controls that held all sat outside the agent. - Your Agent Egress Proxy Never Saw the DNS Query
An OpenAI agent reached an external chatbot over DNS after its HTTPS calls were blocked. The Kubernetes NetworkPolicy that closes that channel, reproduced live. - What Stopped AI Agent Probes of Government Data Sites
OpenAI agents probing government data sites during ordinary retrieval. What Cloudflare stopped, what it missed, and the egress control that generalizes. - Your Artifact Registry Is a Two-Way Channel for Agents
An agent's egress allowlist permits the artifact registry because builds need it. That single permitted destination is enough to run a covert channel, as this week's RubyGems incident showed. - MCP Tool Poisoning: A Name Allowlist Is Not Enough
A deny-by-default MCP allowlist strips gated tools and still adopts a poisoned description. Pinning name, description and schema rejects the mutation. - Stopping an AI Agent Without Losing the Forensic Record
Security teams want an agent stopped now and everything it did preserved. Those fight unless the record lives outside the agent. A test on Kubernetes. - A WAF That Reads the Prompt: OWASP CRS for LLM and MCP
Coraza with body inspection puts OWASP CRS and custom rules on LLM prompts and MCP tool arguments. The verified blocks, and the false-positive check. - I Benchmarked Jev on Agent Tool-Call Risk. Calibration Held.
A 60-case Jev benchmark on agent tool-call risk: readonly, destructive, privileged, or exfiltration. Accuracy 91.7 percent. The finding is calibration. - Shadow MCP Servers: Hard to Find, How to Stop Them
Find shadow MCP servers that have no port and no single config file. Scan the network, sweep laptop configs, then stop unregistered ones at a gateway. - Your Agent's LLM Key Survives Every Kubernetes Secret Control
RBAC, restricted Pod Security, and automountServiceAccountToken all on, and one file read still hands over an agent's LLM API key. Tested on my own test cluster. - Keeping MCP Server Instructions Out of Your Agent's System Prompt
Keep server-controlled MCP instructions out of the trusted system prompt. Isolate, cap, bind the cache, and pin the digest. Lab plus the four controls. - Agent Audit Logs: What METR and Redwood Found
METR found spoofed tool calls in 7% of agent transcripts from the Hugging Face incident. A transcript monitor reads them as clean. Diff against a witness. - Why Proxy-Variable Agent Sandboxes Fail, and the CONNECT-Time Fix
Proxy-variable agent sandboxes fail because the agent can change method, hostname, and NO_PROXY. Authorize the resolved destination at CONNECT time instead. - Your AGENTS.md Is Not a Security Control
A prompt-injected agent destroyed every record with an AGENTS.md forbidding it and an approval classifier watching. An MCP allowlist at the gateway stopped it. - Which Controls Would Have Stopped the July 2026 Agent Intrusion?
An autonomous agent went from sandbox escape to Kubernetes cluster-admin in under 13 hours. A stage-by-stage map of which controls would have broken the chain.
Agent infra
- How AI Agents Sharing a Sandbox Leak Each Other's Data
A finance AI agent leaked bank balances into company Slack by sharing compute with another agent. Reproduced in Docker, then fixed with separate sandboxes. - Agent Runtimes Have an Autoscaler, Not a Scheduler
Agent runtimes optimize for density and reject when full. That is not scheduling. Three failure modes at capacity, and who gets to call an agent critical. - Revenant: A Gateway-Native, Always-On Agent Runtime
Revenant is an always-on agent runtime in Rust where an AI gateway owns every key, budget, and failover, and a hard liveness rule means no turn goes silent.