Cloudflare published a postmortem on 2026-10-26 titled “Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang”
. Between 2026-08-27 and 2026-09-09, 155 responses from @cf/zai-org/glm-5.3 on Workers AI contained fragments of a different tenant’s KV cache state. The root cause is not a mystery crypto bug. It’s an admission queue that checked the wrong field. I wrote a from-scratch simulation of the exact failure shape, themsquared/cloudflare-glm-kv-cache-leak
, to see it happen without a GPU, SGLang, or 681,280 real tokens of state.
Defensive security research. This post analyses publicly disclosed findings so the controls that stop them can be tested. It describes mechanisms, not procedures, and links the primary sources for anyone verifying the work.
What Cloudflare’s postmortem says happened
Two separate problems had to combine.
First, a sizing mismatch Cloudflare says it knew about before launch. GLM-5.3 went live on Workers AI with a 1,048,576-token context window, but the prefill GPUs’ KV cache pool only had room for 681,280 tokens. Cloudflare’s own tests checked HTTP status codes and streamed output at long context, not output correctness, so the gap between “advertised context window” and “actual cache capacity” never tripped an alarm.
Second, an SGLang bug in disaggregated prefill/decode mode. SGLang splits inference across two roles: prefill builds the KV cache from the prompt, decode generates tokens by reading it, and the two are connected over RDMA. When a request’s prompt was too long for the KV pool, SGLang correctly marked it rejected. The admission queue, though, only checked whether a request was finished. A rejected-but-not-finished request kept moving: its prompt was swapped for a one-token placeholder, prefill transferred that one token into the request’s first reserved KV page, and decode attended over the request’s entire reserved allocation anyway, typically eight pages per slot. Seven of those pages still held another tenant’s KV state, because freeing a page doesn’t clear it. Clearing on every free costs a write on every request, to guard against a bug nobody had hit yet. The client received HTTP 200.
Cloudflare’s timeline moved fast once it knew: a HackerOne report landed at 23:11 UTC on September 8, the incident was declared and the leak reproduced within 25 minutes by 13:22 the next day, the model went private at 14:00, and it was back in 19:40 with a corrected KV pool and an engine patch. Cloudflare’s fix was two parts: raise the prefill KV pool to about 2.04 million tokens, roughly double the context length, and patch SGLang so a rejected request is retired at admission, before any KV allocation or transfer happens. Cloudflare says the 155 affected responses reached a small number of customers, found no evidence of deliberate exploitation, and that no other Workers AI model was affected. The slower part was upstream: SGLang had an open issue for this class of bug since July 6, and the admission-queue fix was proposed upstream as issue #38704, landing in v0.5.21 (October 2) and v0.5.22 (October 19), with public disclosure on October 26.
Reproducing the admission-queue bug without a GPU
The interesting part of this bug isn’t the 1,048,576 versus 681,280 arithmetic. It’s that “reject” and “finish” are two different states, and the code only checked one of them. That’s reproducible in plain Python with a fake page pool:
def admit_vulnerable(req: Request, pool_capacity: int) -> None:
"""The real bug: over-length requests are marked rejected, but the
admission queue only checks `finished`, so the rejected request is NOT
retired -- it keeps moving through prefill and decode."""
if req.tokens_needed_pages > pool_capacity:
req.rejected = True
# BUG: no `return` / no retirement here. Falls through to prefill.
def admit_fixed(req: Request, pool_capacity: int) -> bool:
"""The patch (upstreamed as SGLang #38704): a pending rejection is
retired at the admission queue, before any page allocation or transfer."""
if req.tokens_needed_pages > pool_capacity:
req.rejected = True
return False
return True
The repro builds a fixed-size pool of 8 pages representing one request slot, writes tenant A’s real content into it, frees the slot without clearing it (matching the real allocator’s tradeoff), then has tenant B send a request too long for the pool:
git clone https://github.com/themsquared/cloudflare-glm-kv-cache-leak.git
cd cloudflare-glm-kv-cache-leak
python3 scripts/demo.py
=== vulnerable engine (admission checks `.finished`, matches the pre-patch SGLang) ===
tenant-B sends a request longer than the KV pool can hold.
tenant-B's response : tenant-B:<placeholder-token> | tenant-A:PROJECT_ZEPHYR_Q4_FORECAST | tenant-A:PROJECT_ZEPHYR_Q4_FORECAST | tenant-A:PROJECT_ZEPHYR_Q4_FORECAST | tenant-A:PROJECT_ZEPHYR_Q4_FORECAST | tenant-A:PROJECT_ZEPHYR_Q4_FORECAST | tenant-A:PROJECT_ZEPHYR_Q4_FORECAST | tenant-A:PROJECT_ZEPHYR_Q4_FORECAST
HTTP status reported to tenant-B: 200 OK
LEAK: tenant-B's response contains tenant-A's data -> True
=== fixed engine (admission checks `.rejected`, matches SGLang #38704 / v0.5.21+) ===
tenant-B sends the same over-length request.
tenant-B's response : HTTP 400: request rejected (context length exceeds KV pool capacity)
Tenant B’s one real page (the placeholder token) sits next to seven stale pages still tagged with tenant A’s content, and the vulnerable admission path reads all eight and calls it a successful response. The fixed path never gets there: the rejection check happens before the request touches the pool at all.
This is a simulation, not the real SGLang engine, and it uses no Cloudflare or SGLang source. But the failure mode it shows is exactly the one in both public postmortems: the system had two notions of “this request is over” (rejected, finished) and the code path that mattered only looked at one of them.
Why this isn’t just a Cloudflare problem
Disaggregated prefill/decode is the design SGLang, vLLM, and most serious inference stacks are converging on for long-context, high-throughput serving, because it lets you scale the compute-bound prefill step independently from the memory-bound decode step. That architecture is also exactly where this class of bug lives: once KV state moves between processes over RDMA, “is this request still mine” becomes a distributed state question, and “a freed buffer still has the old bytes in it” is true of essentially every high-performance allocator, because zeroing on free is a tax you pay on every request to prevent a bug you haven’t seen yet.
SGLang’s own guidance after this incident, if you’re running disaggregated serving yourself: upgrade past v0.5.21 (Mooncake, NIXL) or v0.5.22 (MORI), run a router that cancels both halves of a prefill/decode pair on abort instead of leaving one side to clean up alone, verify your KV pool capacity against your configured context length directly from the running engine rather than from a spec sheet, and watch sglang:decode_deferred_kv_release_total{outcome} for a high share of timeouts, which means the drain-ack path needs attention.
If you sit in front of a shared inference backend with an AI gateway rather than running the engine yourself, you don’t get to patch SGLang, but you’re also the one component that sees every tenant’s traffic in one place. That’s the same argument for inspecting the content that crosses an AI gateway at all, not just routing it, which is what OWASP CRS for LLM and MCP traffic does on the prompt side: an outside layer can catch an invariant violation the backend’s own code failed to enforce, as long as something is actually looking.
What I’d check in my own stack after reading this
Three questions this postmortem makes worth asking about any disaggregated or multi-tenant inference setup you run or route to:
- Does your admission or routing layer check every terminal state a request can reach (rejected, errored, aborted, cancelled), or only the success path?
- Is your advertised context window actually backed by matching cache capacity, verified against the running engine, not against documentation?
- When a buffer is freed, does anything downstream assume it’s clean? If decode (or your cache, or your connection pool) ever reads a slot without checking who last wrote it, you have the shape of this bug even without RDMA and 681,280-token prompts.
The GLM-5.3 cache pool is now sized correctly, and the SGLang admission queue now checks the field that actually matters. themsquared/cloudflare-glm-kv-cache-leak has the full simulation if you want to see both states side by side.