# Serve Markdown to Agents at the Gateway, Not the Origin

> An agentgateway route serves markdown to agents on the Accept header while the HTML-only origin stays untouched, plus the two ways a naive version fails.

- Canonical URL: https://webofmike.com/markdown-for-agents-at-the-gateway/
- Author: Mike Moore (https://webofmike.com/about/)
- Published: 2026-09-24
- Last modified: 2026-09-24
- Tags: AI Gateways, AI Agents, Platform Engineering, Kubernetes
- Cite as: Mike Moore, "Serve Markdown to Agents at the Gateway, Not the Origin", Web of Mike (webofmike.com), 2026-09-24. https://webofmike.com/markdown-for-agents-at-the-gateway/


Agents parse HTML badly and expensively. The usual fix is to make the site emit markdown: add a per-page `index.md`, publish an `llms.txt`, change the build. That works when you own the site. It does not work for the docs site you inherited, the vendor knowledge base, or the internal wiki nobody will redeploy this quarter.

So I put the decision in the gateway instead. One origin that serves HTML and only HTML, one 200-line shim, and an agentgateway route that picks between them on the `Accept` header. The origin is not modified and does not know any of it happened. Code at [themsquared/agent-content-negotiation](https://github.com/themsquared/agent-content-negotiation).

The interesting part is not that it works. It is the two ways the naive version fails quietly, and a third failure I shipped into my own converter and did not notice.

## How the routing works

The gateway matches a regex against the request's `Accept` header. Requests that mention `text/markdown` go to the shim, which fetches from the origin and converts. Everything else goes straight to the origin. `Vary: Accept` is set so caches do not serve one variant to the other audience.

Five scenarios against a single URL, changing nothing but the header:

```
SCENARIO 2  an agent that asks  ->  Accept: text/markdown
  PASS  served the markdown variant
  PASS  content-type is text/markdown
  PASS  body is a markdown heading
  PASS  code fence survived
  PASS  link became a markdown link
  PASS  no HTML doctype leaked
  PASS  markdown body contains no HTML
  PASS  site chrome (nav/script/footer) was dropped

SCENARIO 5  payload size, same page, same origin
  html       894 bytes
  markdown   420 bytes
  PASS  markdown variant is smaller (474 bytes less)

assertions: 15 passed, 0 failed
```

That size difference is the whole economic argument in one line. Same page, same origin, less than half the bytes, and none of the remaining bytes are nav, script tags or footer boilerplate that an agent has to read and discard.

## Gotcha one: your agents are asking for `*/*`

This is the one that will actually bite you.

```
SCENARIO 3  THE GOTCHA  ->  Accept: */*   (curl's default, and a lot of agent HTTP clients)
  PASS  served the HTML variant
```

`*/*` does not mention markdown. The agent route does not match. The agent gets the HTML site and a parsing problem it will blame on your docs.

What makes this bad is that it is invisible from the server side. There is no error, no 406, nothing in the logs that looks wrong. The request is well formed and the response is a valid 200. You will conclude the feature works, because when *you* test it you will type `Accept: text/markdown` by hand, and your agents never will.

If you deploy this, the first thing to do is not to celebrate the markdown route. It is to look at the distribution of `Accept` headers you are actually receiving and find out how much of your agent traffic is wildcards.

## Gotcha two: a header regex is not content negotiation

```
SCENARIO 4  THE LIMITATION  ->  Accept: text/html, text/markdown;q=0.1
  PASS  served the markdown variant anyway
```

The client said it would much rather have HTML. `q=0.1` on markdown is close to "only if you have nothing else." A route match cannot read that, because a regex tests for presence, not preference.

This is a routing match, not RFC 9110 proactive negotiation, and the distinction matters the moment a well-behaved client sends a weighted header. It is a real limitation of doing this at the route layer rather than in an application that parses the header properly. Worth knowing before you tell people the gateway does content negotiation, because it does not. It does routing that resembles it.

## The bug I shipped into my own converter

The shim strips site chrome: nav, script, footer. It did this by opening a skip region when it saw a chrome tag and closing it on the matching end tag.

`<link>` is a void element. It has no end tag. So the skip region opened and never closed, the converter consumed the rest of the document, and it returned an **empty document under HTTP 200** with `Content-Type: text/markdown` and one newline of body.

Every status-code check passed. The content type was right. The response was fast. An agent consuming this gets a page that exists and says nothing, and the most likely outcome is that it concludes your documentation is empty rather than that your gateway is broken.

Two lessons, and the second is the general one:

1. Separate container tags from void elements when you walk HTML. `<link>`, `<meta>`, `<img>`, `<br>`, `<hr>` and friends never close.
2. **Assert on body content, not on status.** A 200 with the right content type is not evidence that anything was served. This is the same shape as the false-pass I hit in a [different demo the same week](https://webofmike.com/mcp-tool-poisoning-pin-definitions/), where an assertion on the absence of a string passed against a 404 page. Both bugs are a test that cannot distinguish success from a specific kind of nothing.

## A smaller one worth ten minutes of your life

agentgateway lowercases response header names that you inject in config. `X-Served-Variant` comes back as `x-served-variant`. HTTP header names are case-insensitive so nothing is wrong, but my first assertion pass grepped for the capitalised form and failed on a response that was completely correct. Compare header names case-insensitively in tests.

## Running it

Docker with Compose v2, nothing else. The shim is Python standard library. Validated on Docker 29.7.2 and Compose v5.4.0, arm64 macOS, against agentgateway v1.5.0 and `python:3.12-slim`.

```bash
git clone https://github.com/themsquared/agent-content-negotiation
cd agent-content-negotiation
docker compose up -d --build
./scripts/demo.sh
```

Expected: `assertions: 15 passed, 0 failed`.

```bash
docker compose down
```

I re-ran the whole suite while writing this post rather than quoting the README, so every number above is from a run on the day of publication.

## What this is not

It is not RFC 9110 content negotiation, per scenario 4. It is not a general HTML-to-markdown converter either; the shim is deliberately small enough to read in one sitting, and a real deployment would put something more capable behind the same route.

What the pattern gives you is the ability to make an origin you do not control agent-readable at the traffic layer, and to see what your agents are actually requesting while you do it. Those two things are worth more together than separately, because the header distribution is what tells you whether the markdown route is doing anything at all.

For the gateway side of the same product, [per-key LLM budgets](https://webofmike.com/agentgateway-per-key-llm-budgets/) covers the other thing I keep wanting at this layer: limits that live with the caller's identity rather than the route.

The repo is [themsquared/agent-content-negotiation](https://github.com/themsquared/agent-content-negotiation), Apache-2.0, and the demo takes about two minutes.