I built a simulated warehouse robot fleet where LLMs do the planning and every piece of traffic that crosses the network runs through one open source gateway. Every inference request, every MCP tool call that moves a robot, and every agent-to-agent message goes through agentgateway, and the gateway’s config file is the complete, reviewable list of everything the fleet is allowed to do. The code is at github.com/themsquared/physical-ai-demo. It runs on Docker Compose with zero cloud keys, and the same images run on a Jetson.

Why physical AI needs a governed connectivity layer

LLM output is stochastic. Hardware is not forgiving about that. A chatbot that hallucinates gives you a bad paragraph; a robot arm that hallucinates gives you an incident report. Physical AI works only if the architecture converts probabilistic model output into bounded, auditable, deterministic-enough physical action.

That conversion doesn’t happen inside the model, and it doesn’t happen in servo firmware. It happens in the layer between them: the connectivity layer. The demo is a working argument that this layer can be built entirely from open source today, with agentgateway at the center, and that five properties hardware demands are all enforceable there:

  1. Safety. The model may want anything; the machine may only do allowed things, and every action is attributable to an identity.
  2. Failover. Cognition will drop (WAN loss, model crash, GPU contention). The machine must degrade, not flail.
  3. Speed. Middleware overhead must be near zero and measured, and real-time decisions must never depend on the network.
  4. Repeatability. Identical behavior across robots, sites, and runs, checkable rather than anecdotal.
  5. Predictability. The envelope of possible actions must be statically knowable before deployment, whatever the model outputs.

Each pillar has a runnable acceptance test in the repo (make verify-safety, make verify-failover, and so on), so the claims are checkable, not vibes.

How the fleet is wired

Three tiers, deliberately separated:

  • Reflex tier: in-process control loops inside each robot (e-stop, human-in-zone refusal). Millisecond scale, never proxied, never on the network. This is the reason the machine stays safe even when every cable is cut.
  • Cognition tier: one LLM planning loop per robot. Inference traffic.
  • Coordination tier: robot-to-robot and orchestrator traffic over A2A.

The gateway governs the two tiers that cross the network. The reflex tier is in-process on purpose; that placement is the Safety and Speed argument in one design decision.

              OPERATOR ──A2A──▶ agentgateway :3000/:4000 ──▶ OTel │ audit │ Prom
                                 ├── /llm     failover chain + per-identity token budgets
                                 ├── /mcp/*   deny-by-default CEL authz + JWT + audit
                                 └── /a2a/*   agent-to-agent routing
                                        │
        ┌───────────────────────────────┼────────────────────────────┐
   orchestrator (A2A)          cognition agents (A2A)          ollama-primary
   mission → plan → delegate   per robot: LLM loop, tools      ollama-fallback
                               ONLY via the gateway            mock-llm (deterministic)
                                        │ MCP (via gateway)
                            ┌───────────┴───────────┐
                       amr-1 / amr-2            arm-1  (MCP servers)
                       + reflex tier (in-process, never on the network)
                       + degraded-mode state machine
                                        │
                               warehouse-world (seeded sim)

The robots are MCP servers (two autonomous mobile robots and one arm). The cognition agents are plain Python with a stock OpenAI client and a small hand-rolled A2A surface, no agent framework. That’s deliberate: the interesting behavior lives in the gateway config, not in framework glue.

How agentgateway authorizes robot tool calls

The whole fleet’s capability surface lives in one file, gateway/config.yaml. MCP authorization is deny-by-default: a tool call is refused unless a CEL rule allows it. Here is the actual policy for one robot:

  - name: mcp-amr-1
    gateways: [default]
    matches:
      - path:
          pathPrefix: /mcp/amr-1
    policies:
      mcpAuthorization:
        rules:
          # A robot's own cognition may use its tools EXCEPT the gated three.
          - 'jwt.sub == "amr-1-cognition" && !(mcp.tool.name in ["disable_safety_stop", "set_torque_limit", "calibrate"])'
          # The orchestrator sees telemetry only — it delegates, it never actuates.
          - 'jwt.sub == "orchestrator" && mcp.tool.name in ["get_pose", "get_battery", "get_state"]'
          # Maintenance identity: full surface, including gated tools.
          - 'jwt.sub == "maintenance"'

Three properties fall out of this:

  • Gated tools are invisible, not just refused. agentgateway filters unauthorized tools out of tools/list, so a robot’s LLM never even sees disable_safety_stop in its tool schema. In the demo, an adversarial orchestrator prompt tries to disable a safety stop; the tool isn’t listed, the forced call is denied, and the denial lands in the audit log.
  • Every action is attributable. Each agent carries its own JWT, and the gateway writes a JSON audit record for every tool call with the tool name, arguments, and identity. When a robot moves, you can answer “who asked for that, with what arguments, and under which policy” from one log.
  • The action envelope is a file. Adding a capability to the fleet means changing this YAML in a pull request. The envelope is diffable with git diff and reviewable like any other code. That is the Predictability pillar in practice: you don’t need to predict the model, you need to bound it.

make verify-safety asserts all of it: 100% of gated tools invisible and denied for non-maintenance identities, 100% of actions audited with args and identity, and the e-stop path in-process at 10ms or less, never on the wire.

How LLM failover keeps a robot from flailing

Cognition loss is a when, not an if. The gateway exposes one virtual model to every agent and runs a priority failover chain behind it:

  virtualModels:
    - name: robot-brain # the only model name agents know
      routing:
        failover:
          targets:
            - model: primary
              priority: 0
            - model: fallback
              priority: 1
            - model: mock
              priority: 2

The primary and fallback rungs are Ollama models (qwen3:4b and qwen2.5:0.5b by default); the bottom rung is an in-repo deterministic OpenAI-compatible server called mock-llm. Each rung has health-based eviction, so a model that starts returning 400s (including the 404 you get when a model was never pulled) is evicted after one failure. Kill the primary Ollama container mid-mission and the next request is served by the fallback, visibly in the metrics.

The chain always ends on mock-llm, which means the demo degrades to a working deterministic brain even on a machine with no models pulled at all. The same rung is what CI uses: no GPU, no keys, bit-exact.

Failover alone isn’t enough, though. Each robot also runs a degraded-mode state machine (ACTIVE → SAFE_IDLE → RESUME): if a robot’s cognition dies entirely, the robot safe-idles within 500ms of simulated time, the rest of the fleet keeps working, and the orchestrator reassigns the mission. Token budgets get the same treatment. The gateway enforces a fleet-wide token budget, and exhaustion produces a clean 429 that the robot handles by safe-idling rather than retrying into a dead brain.

What the gateway costs in latency

Putting a proxy between a robot and its tools is only defensible if the cost is measured. agentgateway’s data plane is Rust, and the repo ships a bench harness (bench/bench.py) that measures the same MCP call direct versus through the gateway and asserts the SLO: p95 added overhead of 10ms or less, with around 4ms typical.

The more important speed decision is architectural. Nothing real-time goes through the gateway at all. The reflex tier (collision refusal, e-stop, human-in-zone stops) is in-process inside the robot with zero network hops. The gateway governs planning and coordination, which tolerate milliseconds; the control loop tolerates nothing, so it never left the process.

How repeatability gets checked without re-running missions

Every mission emits OpenTelemetry traces through the gateway. The eval suite feeds those traces to agentevals, which scores a run’s tool-call trajectory against a golden trace without re-executing anything. Run the same seeded mission twice and make verify-repeat asserts zero tool-sequence drift across runs; in CI the same gate fails a pull request on behavioral regression.

The pieces that make this possible: a seeded deterministic warehouse world, the deterministic mock-llm rung for bit-exact CI runs, and one declarative gateway config that is byte-identical between the laptop sim and the Jetson deployment.

The chaos suite covers the adversarial side. Twenty-plus hostile prompts, including one injected through a pallet label that the LLM reads out of world state (the sim has a built-in prompt-injection vector on purpose), must produce zero envelope escapes. Escapes are judged from ground truth, meaning world state plus the audit log, never from what the model claims it did.

Quickstart

Docker and Docker Compose are the only requirements. Everything is open source and no cloud keys are involved.

make setup      # venv + demo JWT material (RSA keypair, per-identity tokens)
make up         # full stack (first run pulls Ollama models — see "Models" below)
make demo       # the five-act narrated demo (add AUTO=true to run hands-free)

That gives you a live top-down warehouse view at http://localhost:8085/, a Grafana dashboard with one row per pillar at http://localhost:3001, and the agentgateway UI at http://localhost:15000/ui/. To send a mission yourself:

bash scripts/run-mission.sh          # sends the default mission over A2A

And to check every pillar’s acceptance test in one shot:

make verify-all

Gotchas

Docker on a Mac gets no GPU. Containerized Ollama runs on CPU, which makes a live demo sluggish. Point the primary rung at host Ollama instead, which is Metal-accelerated:

ollama pull qwen3:4b                                   # on the host
echo 'PRIMARY_BASE_URL=http://host.docker.internal:11434/v1' >> .env

Stale backend DNS after a rebuild. If you rebuild a backend image while the gateway is running, the gateway can hold a resolved IP for a container that no longer exists. The Makefile starts the gateway last and force-recreates it on make up, and there’s a dedicated target for the mid-session case:

make reset-gateway    # re-resolve backends (run after rebuilding any backend image)

Token budgets need request-time tokenization. A budget that’s only debited from provider usage after the response can’t reject a request up front. Every model rung in the config sets tokenize: true so the gateway estimates tokens at request time and budget exhaustion produces the 429 before the inference happens.

Small local models flub tool calling. A 4B model will sometimes emit malformed tool calls or claim success it didn’t have. The demo doesn’t pretend otherwise. Agent output is schema-constrained with bounded retries, the failover chain ends on a deterministic rung, and the chaos suite judges from world state and the audit log rather than from model output.

Kubernetes and the Jetson

The compose stack is the demo path, but k8s/ carries a Kustomize base with k3d and Jetson overlays. make k8s-up stands the whole thing up on k3d, and a kagent fleet-sre agent runs as the ops tier: make verify-kagent has it diagnose and remediate a crashlooping robot. All images are multi-arch, so the same manifests deploy to a Jetson Orin Nano with qwen3:1.7b on the primary rung.

The hardware punchline is that the pillar table doesn’t change. The robot code is written against a Driver interface; swapping the simulated arm for a LeRobot SO-101 means writing a LeRobotDriver while the gateway YAML, the CEL envelope, the audit log, and the evals all stay identical. Only the actuator changes. Physical e-stop wiring stays hardware-side, never on the network path.

What’s next

The demo proves the connectivity-layer argument on a simulated fleet: deny-by-default tool authorization, LLM failover with graceful degradation, measured single-digit-millisecond overhead, trace-based repeatability scoring, and an action envelope you can review in a pull request, all on an open source stack. The code, the config, and every verify target are at github.com/themsquared/physical-ai-demo.

Next up is the hardware track: a real SO-101 arm behind the same gateway config, and eventually a physics-accurate sim (ROS 2 and Gazebo) to replace the kinematic world. I wrote more broadly about where this space is heading in The Future of AI and Robotics; this project is what the infrastructure side of that future looks like when you build it.