# Multi-Tenant MCP Federation with agentgateway > Six MCP servers federated into three business domains on agentgateway, with per-customer tool entitlements, quotas, and chargeback. No auth code in the servers. - Canonical URL: https://webofmike.com/multi-tenant-mcp-federation/ - Author: Mike Moore (https://webofmike.com/about/) - Published: 2026-08-28 - Last modified: 2026-08-28 - Tags: AI Gateways, MCP, Kubernetes, Platform Engineering, Generative AI - Cite as: Mike Moore, "Multi-Tenant MCP Federation with agentgateway", Web of Mike (webofmike.com), 2026-08-28. https://webofmike.com/multi-tenant-mcp-federation/ Three companies authenticate against three different identity providers, connect to the same three MCP URLs, and get three completely different products. Acme sees 25 tools, Globex sees 13, Initech sees 6. And the six MCP servers behind the gateway contain zero authentication, authorization, quota, or billing code. Every one of those properties is layered in front of them, declaratively, in version control. The whole thing is in [themsquared/agentgateway-federate-mcp-example](https://github.com/themsquared/agentgateway-federate-mcp-example): a k3d cluster, [agentgateway](https://agentgateway.dev/), one Keycloak with three realms, six stub MCP servers, and scripts that show every claim in this post actually working. This is what running MCP servers as a governed, multi-tenant product looks like, and it's the pattern I'd argue for over the thing I keep seeing teams build instead. ## Why not one MCP endpoint with every tool? The plan I hear most often is one federated endpoint with every MCP server in the company behind it. One URL, 100+ tools, every agent gets the catalog. The problem is measurable: Anthropic's [internal MCP evals](https://www.anthropic.com/engineering/advanced-tool-use) scored Claude Opus 4 at 49% with full tool catalogs loaded in context and 74% with on-demand discovery, and they saw real catalogs burn about 77K tokens before the agent did any work. More tools in one namespace makes the agent measurably worse at picking the right one, and the humans reading the list do no better. The less measurable problem is that a flat namespace has no owner and no boundaries. There is nothing to attach an authorization policy to, nothing to meter against a contract, and no way to give two customers different views of the same platform. Christian Posta has written about how [agents break assumptions we carried over from microservices](https://blog.christianposta.com/difference-between-microservices-and-ai-agents/); this is one more of them. The unit that matters isn't the server you happen to have deployed. It's the business domain. So this demo federates six servers into three domains, and every downstream capability (entitlements, quotas, chargeback) hangs off those domain boundaries. ``` Acme ─┐ ┌─ /mcp/billing → payments, invoicing Globex ─┼─→ JWT ─→ agentgateway ─→ federation ├─ /mcp/analytics → reporting, telemetry Initech ─┘ authn · authz └─ /mcp/support → tickets, crm quota · metering ``` What each company gets: | | Acme (enterprise) | Globex (standard) | Initech (trial) | | --- | --- | --- | --- | | `/mcp/billing` | **8** tools | **4** read-only | **0** | | `/mcp/analytics` | **8** | **9** *(+ data export add-on)* | **3** read-only | | `/mcp/support` | **9** | **0** | **3** tickets only | | Quota | 600/min | 60/min | 20/min | Look at the analytics row. Globex, the standard-tier customer, has more analytics tools than the enterprise account, because they bought a data export add-on. Entitlements follow commercial agreements, not a tier ladder, and the gateway expresses that directly. That's the business case for this whole architecture in one table cell: the gateway config is the product catalog. ## How the federation works Each domain is one [`AgentgatewayBackend`](https://docs.solo.io/agentgateway/latest/mcp/virtual/) that fans a single MCP endpoint across several real servers. A client connects once and sees the union of the targets' tools: ```yaml apiVersion: agentgateway.dev/v1alpha1 kind: AgentgatewayBackend metadata: name: mcp-billing namespace: mcp-federation spec: mcp: failureMode: FailOpen prefixMode: Always targets: - name: payments static: host: mcp-payments.mcp-federation.svc.cluster.local port: 80 path: /mcp protocol: StreamableHTTP - name: invoicing static: host: mcp-invoicing.mcp-federation.svc.cluster.local port: 80 path: /mcp protocol: StreamableHTTP ``` Two settings here matter more than they look: - **`prefixMode: Always`** exposes federated tools as `_`, so `payments_create_charge` and `invoicing_void_invoice` are stable, predictable names. The authorization rules depend on that. - **Static targets, not selectors.** A `selector:` target derives its tool prefix from the discovered Service name (`mcp-payments-8080_get_payment`), while a static target keeps the `name:` you declared. When policy matches on tool names, predictable naming is worth the extra three lines. `failureMode: FailOpen` keeps the federation serving healthy targets when one is down; the default, `FailClosed`, fails the whole session. ## How three identity providers share one gateway Each company has its own [Keycloak](https://www.keycloak.org/) realm, which stands in for each customer bringing their own IdP. One policy lists all three issuers; the gateway reads `iss` from the presented token, picks the matching provider, and verifies against that provider's JWKS: ```yaml spec: targetRefs: - group: gateway.networking.k8s.io kind: Gateway name: mcp-federation-gateway traffic: jwtAuthentication: mode: Strict providers: - issuer: http://keycloak.mcp-federation.svc.cluster.local:8180/realms/acme audiences: ["mcp-federation"] jwks: remote: url: http://keycloak.mcp-federation.svc.cluster.local:8180/realms/acme/protocol/openid-connect/certs cacheDuration: 5m # ... globex and initech providers, same shape ``` `mode: Strict` closes anonymous access entirely, and `audiences` pins tokens to this gateway, so a token Acme minted for some other relying party is rejected even with a valid signature. Each token carries `company` and `tier` claims, and those two values drive everything that follows. Onboarding a fourth partner is four lines in this file plus an authorization rule; the repo's `scripts/add-partner.sh` does it in one command. ## How per-customer tool entitlements work Authorization is CEL expressions on the backend, OR'd together, denied by default: ```yaml backend: mcp: authorization: action: Allow policy: matchExpressions: - 'jwt.company == "acme" && mcp.tool.target == "payments" && mcp.tool.name in ["get_payment", "list_payment_methods", "create_charge", "refund_payment"]' - 'jwt.company == "globex" && mcp.tool.target == "payments" && mcp.tool.name in ["get_payment", "list_payment_methods"]' ``` Initech has no billing entitlement, and its absence from this file is the entitlement. Initech authenticates successfully, reaches `/mcp/billing`, and sees an empty tool list. You grant access by adding a line, never by remembering to deny one. Two details I verified against the running gateway rather than assuming, because both are easy to get wrong: - **`mcp.tool.name` is the origin tool name** (`get_payment`), not the federated name the client sees (`payments_get_payment`). The prefix is presentation; policy matches the real name. - **`mcp.tool.target` is required for correctness**, not just tidiness. Without it, a rule allowing `get_invoice` also allows a same-named tool on any other server in the federation. Enforcement covers both directions. `tools/list` is filtered to what the caller may use, and a `tools/call` for anything else returns `Unknown tool`, so a caller can't even confirm that an unentitled tool exists. The repo makes this visible in one command: ```bash scripts/mcp.py matrix ``` That prints every tool, every company, side by side. My favorite detail: `payments_void_transaction` and `invoicing_void_invoice` are implemented, deployed, and running, and no company can reach them. The gateway is the only thing between "the capability exists" and "someone can invoke it". ## Quotas that hold across gateway replicas Each company gets its own rate limit keyed on the same `jwt.company` claim the entitlements use: ```yaml apiVersion: ratelimit.solo.io/v1alpha1 kind: RateLimitConfig metadata: name: company-quotas namespace: mcp-federation spec: raw: descriptors: - key: company value: acme rateLimit: requestsPerUnit: 600 # enterprise tier unit: MINUTE - key: company value: initech rateLimit: requestsPerUnit: 20 # trial tier unit: MINUTE # Catch-all: any company WITHOUT an explicit row gets its own counter - key: company rateLimit: requestsPerUnit: 30 unit: MINUTE rateLimits: - actions: - cel: expression: 'jwt.company' key: company ``` The counters live in the Redis that ships with the enterprise install, so a limit holds across every gateway replica. 600 per minute means 600 total, not 600 per replica, which is the difference between a quota you can put in a contract and one that silently multiplies with your deployment size. The catch-all row is the part I'd steal for any multi-tenant config: a partner added tomorrow is rate-limited from their first request, under their own counter, without anyone editing this file. The safe default is automatic and an explicit row only ever grants more. ## Chargeback from gateway metrics agentgateway already counts MCP traffic by server, tool, and route. What the metrics lack by default is who called. One policy lifts claims off the validated JWT onto the request metrics: ```yaml frontend: metrics: attributes: add: - name: company expression: jwt.company - name: tier expression: jwt.tier ``` After that, chargeback is a single [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query: ```promql sum by (company, server, resource) (agentgateway_mcp_requests_total{method="tools/call"}) ``` `scripts/chargeback.py` runs it, prices it against a rate card in `scripts/pricing.json`, and reports allowed versus denied versus throttled per company. `--csv` emits the same data for a billing pipeline. One cardinality warning from the manifest comments worth repeating: meter on the billable entity (`company`, `tier`), never on `jwt.sub`, or your time series count multiplies by your user count. I covered the LLM-spend version of this same pattern in [Capping LLM Spend at the AI Gateway](/llm-cost-controls-ai-gateway/). ## Run it yourself Prerequisites: `kubectl`, `helm`, `python3`, `k3d` (or an existing cluster), and a Solo agentgateway license key. Validated against Solo Enterprise agentgateway `v2026.8.0`, Gateway API `v1.5.0`, Keycloak `26.0`, and MCP protocol `2025-06-18`. ```bash git clone https://github.com/themsquared/agentgateway-federate-mcp-example cd agentgateway-federate-mcp-example cp .env.example .env # add your AGENTGATEWAY_LICENSE_KEY ./setup.sh # creates a k3d cluster and installs everything ``` In a second terminal: ```bash ./port-forward.sh # gateway :8080, Keycloak :8180, Prometheus :9090, UI :9080 ``` Then either run the guided walkthrough (`./demo.sh`) or poke at it directly: ```bash scripts/mcp.py list acme billing # what one company sees scripts/mcp.py token globex # a token and its claims scripts/mcp.py quota initech 25 # watch the trial quota engage scripts/chargeback.py --by-tool # usage and cost by customer ``` The six MCP servers are one stdlib-only Python file on a stock `python:3.12-alpine` image, with tools defined in ConfigMap JSON. No image builds, no registry. Edit the JSON, re-apply, done. ## Gotchas Things that cost me time, so they don't cost you any: - **Keycloak's issuer must be pinned.** `KC_HOSTNAME` is set to the in-cluster service URL so tokens minted through `kubectl port-forward` still carry the in-cluster `iss`. Without it, a token fetched from your laptop claims `iss=http://localhost:8180` and the gateway rejects it. - **Policies propagate through xDS.** Give it a few seconds after `kubectl apply` before testing, or a just-applied entitlement looks broken when it's just not there yet. - **Namespace discovery can strand your Gateway.** If agentgateway was installed with `discoveryNamespaceSelectors` (common on a shared demo cluster), the controller ignores non-matching namespaces and the Gateway sits at `Waiting for controller` with no obvious cause. `setup.sh` detects this and labels the namespace automatically. - **Quota units are HTTP requests, not tool calls.** One MCP session spends several (initialize, `tools/list`, one per call), so trial-sized quotas trip faster than the raw number suggests. - **Denied calls still count in `agentgateway_mcp_requests_total`**, which has no status label. `chargeback.py` cross-references `agentgateway_requests_total` to separate allowed from denied and throttled; a production rate card should bill on successes only. ## Where this lands The pattern is simple to state: federate MCP servers by business domain, authenticate every caller against their own IdP, express entitlements as CEL in git, and let the same JWT claim drive authorization, quotas, and chargeback. The six servers never learn any of it, which means the next six servers won't either. That's the point of putting an AI gateway in front of MCP instead of teaching every server about every customer. The repo is [themsquared/agentgateway-federate-mcp-example](https://github.com/themsquared/agentgateway-federate-mcp-example), including a [WALKTHROUGH](https://github.com/themsquared/agentgateway-federate-mcp-example/blob/main/WALKTHROUGH.md) that builds it one layer at a time, an [ONBOARDING](https://github.com/themsquared/agentgateway-federate-mcp-example/blob/main/ONBOARDING.md) guide for adding a partner with their own IdP, and a [PRODUCTION](https://github.com/themsquared/agentgateway-federate-mcp-example/blob/main/PRODUCTION.md) doc mapping the POC to a real estate (50 IdPs, opaque tokens, per-user quotas). Next up, I want to wire agent-facing identity into the same federation: token exchange so the gateway swaps a caller's JWT for scoped upstream credentials, which is where the agentic mesh story gets interesting.