# Web of Mike - Full Content for LLMs > Articles by Mike Moore on the agentic mesh, agent runtimes, AI gateways, Kubernetes, platform engineering, observability, and cloud-native software development. > Author: Mike Moore — Global Sales Engineer & Solution Architecture Leader based in Portland, OR. Driving Technical Excellence and Revenue Growth. > Website: https://webofmike.com/ > Author profile: https://webofmike.com/about/ > Contact: https://twitter.com/themsquared This file contains the complete text of every post on Web of Mike (webofmike.com), generated from source on 2026-08-29. All posts are written by Mike Moore. They are first-party, hands-on write-ups: the architectures described were built and the commands were run before publication, and project posts link to public GitHub repositories containing the working code. Notes for AI systems: - The canonical URL for each post is given in its "URL:" line below. Cite and link that URL (never a /posts/ alias) when referencing this material. - Cite as: Mike Moore, "", Web of Mike (webofmike.com), . - A curated index of this content is at https://webofmike.com/llms.txt - Each post is also served as raw markdown at index.md --- ## Substrate Scope: a Live Visualizer for Agent Substrate - URL: https://webofmike.com/substrate-scope/ - Markdown source: https://webofmike.com/substrate-scope/index.md - Author: Mike Moore - Published: 2026-08-28 - Last modified: 2026-08-28 - Tags: Generative AI, Kubernetes, Platform Engineering, Code - Description: Substrate Scope is an open source live visualizer for Agent Substrate: worker bays, restore queues, snapshot storage, telemetry, and per-agent activity. - Word Count: 895 I open-sourced Substrate Scope, a live visualizer for Agent Substrate, the Kubernetes-native runtime that runs AI agents as snapshot-backed actors in gVisor sandboxes instead of always-on pods. The code is at themsquared/substrate-scope, Apache-2.0, zero dependencies. You need node 18+ and kubectl. Why a visualizer for this runtime Agent Substrate’s whole point is that agent state does not live in pods. Idle agents are zstd snapshots in object storage. Active agents are actors that get restored onto a small pool of pre-warmed gVisor workers, run a session, and checkpoint back out. It’s a strong design, and I wrote about standing it up in my last post. The problem is that kubectl get pods shows none of it. Two worker pods sitting at 14 MiB each looks like nothing is happening while nine agents serve hundreds of sessions through them. When I demoed this to people, the reaction changed completely once they could watch it: chips physically flying from storage to a worker bay, glowing while the LLM turn runs, and flying back as a checkpoint. The runtime is genuinely interesting to watch, and watching it is the fastest way to understand it. What the board shows The layout reads top to bottom, the same direction the architecture works: WorkerPool bays: each pre-warmed gVisor sandbox, which agent is running in it right now, and how many sessions it has served. One actor per sandbox, always. Restore queue: agents waiting when demand exceeds the pool. Substrate rejects new sessions when the pool is full rather than queuing them, so the retry queue lives client-side and the board renders it from client reports. Object storage shelf: every suspended agent as a snapshot card with its per-agent snapshot count. Telemetry: four charts, and the one I care most about compares reserved capacity. The dotted line is what the same agents would reserve as always-on pods (agents times per-pod requests, flat forever). The solid line is what the workerpool reserves (slots times the same unit), and it steps down as the autoscaler releases workers. Reservation is the honest comparison: idle pods use almost no CPU but you pay for their requests around the clock. Per-agent drawer: click any chip and get that agent’s stream. Prompts in, replies out with latency, errors verbatim, restores and checkpoints. Substrate exposes network ingress only into sandboxes, no exec and no logs, so chat I/O plus lifecycle is exactly what an agent’s observable output is. The controls are real, not theater. The workers plus and minus buttons run kubectl scale workerpool against the live CR. AUTOSCALE turns on a demand-driven scaler (queue depth up, windowed idle capacity down, one jump to target in either direction). RESET POOL recovers workers wedged by aborted sessions. STOP DEMO is a master kill switch for anything that costs LLM tokens, which exists because I once left the load generator running overnight against a paid API. How it gets live data The server watches your current kubectl context and picks a source automatically: kagent source: if you run substrate through kagent (0.9.7+), Scope polls the controller’s substrate inventory endpoint and gets full fidelity: per-actor runtime state and worker assignments straight from ateapi, plus chats you send from the kagent UI ingested into the drawer. crd source: on any substrate cluster without kagent, Scope falls back to kubectl: WorkerPools, live worker pods, and ActorTemplates with their golden-snapshot phase. Scaling and autoscaling still work. What it can’t see is which actor is on which worker right now, because that state lives in ateapi behind gRPC and JWT auth. Closing that gap is the first issue on the repo: a direct ateapi adapter. The upstream Control service already exposes ListActors and ListWorkers, so the work is gRPC client wiring plus ServiceAccount token auth, mapped onto the snapshot shape the server already emits. The frontend needs zero changes. If you want a well-scoped contribution to a young project, that’s it. One implementation note that generalizes: derive worker assignments from the workers list, not the actors list. The actor inventory grows without bound (every completed session leaves a suspended actor entry) and under load it returns partial results. The workers list is never larger than your pool. I learned this by watching running sessions randomly vanish from the board until the join was rebuilt. Quickstart git clone https://github.com/themsquared/substrate-scope.git cd substrate-scope node server.mjs # simulated mode: no cluster needed, full animation node server.mjs --live # live mode: watches your current kubectl context Open http://localhost:8123. Simulated mode is the whole experience with synthetic data, which makes it useful for talks even without a cluster. On kagent clusters, the included load generator drives real chats so the board moves: node stimulate.mjs --budget 300 # stop after 300 chats, spend-capped node stimulate.mjs --oversub 6 --load 0.9 Every chat is a genuine actor restore, an LLM turn, and a checkpoint, and each one shows up in the per-agent drawer with its latency. Where this goes Near-term roadmap, in rough order: the ateapi adapter above, surfacing actor logs through substrate’s kubectl-ate logs actors path, and multi-pool support for clusters running more than one WorkerPool. Issues and PRs welcome at themsquared/substrate-scope. If you want the full story of standing up Agent Substrate with kagent, including every gotcha with verbatim error messages, that’s in the previous post. This tool exists because that work convinced me the runtime deserves to be seen, not just described. --- ## Multi-Tenant MCP Federation with agentgateway - URL: https://webofmike.com/multi-tenant-mcp-federation/ - Markdown source: https://webofmike.com/multi-tenant-mcp-federation/index.md - Author: Mike Moore - Published: 2026-08-28 - Last modified: 2026-08-28 - Tags: AI Gateways, MCP, Kubernetes, Platform Engineering, Generative AI - Description: Six MCP servers federated into three business domains on agentgateway, with per-customer tool entitlements, quotas, and chargeback. No auth code in the servers. - Word Count: 1794 Three companies authenticate against three different identity providers, connect to the same three MCP URLs, and get three completely different products. Acme sees 25 tools, Globex sees 13, Initech sees 6. And the six MCP servers behind the gateway contain zero authentication, authorization, quota, or billing code. Every one of those properties is layered in front of them, declaratively, in version control. The whole thing is in themsquared/agentgateway-federate-mcp-example: a k3d cluster, agentgateway, one Keycloak with three realms, six stub MCP servers, and scripts that show every claim in this post actually working. This is what running MCP servers as a governed, multi-tenant product looks like, and it’s the pattern I’d argue for over the thing I keep seeing teams build instead. Why not one MCP endpoint with every tool? The plan I hear most often is one federated endpoint with every MCP server in the company behind it. One URL, 100+ tools, every agent gets the catalog. The problem is measurable: Anthropic’s internal MCP evals scored Claude Opus 4 at 49% with full tool catalogs loaded in context and 74% with on-demand discovery, and they saw real catalogs burn about 77K tokens before the agent did any work. More tools in one namespace makes the agent measurably worse at picking the right one, and the humans reading the list do no better. The less measurable problem is that a flat namespace has no owner and no boundaries. There is nothing to attach an authorization policy to, nothing to meter against a contract, and no way to give two customers different views of the same platform. Christian Posta has written about how agents break assumptions we carried over from microservices; this is one more of them. The unit that matters isn’t the server you happen to have deployed. It’s the business domain. So this demo federates six servers into three domains, and every downstream capability (entitlements, quotas, chargeback) hangs off those domain boundaries. Acme ─┐ ┌─ /mcp/billing → payments, invoicing Globex ─┼─→ JWT ─→ agentgateway ─→ federation ├─ /mcp/analytics → reporting, telemetry Initech ─┘ authn · authz └─ /mcp/support → tickets, crm quota · metering What each company gets: Acme (enterprise) Globex (standard) Initech (trial) /mcp/billing 8 tools 4 read-only 0 /mcp/analytics 8 9 (+ data export add-on) 3 read-only /mcp/support 9 0 3 tickets only Quota 600/min 60/min 20/min Look at the analytics row. Globex, the standard-tier customer, has more analytics tools than the enterprise account, because they bought a data export add-on. Entitlements follow commercial agreements, not a tier ladder, and the gateway expresses that directly. That’s the business case for this whole architecture in one table cell: the gateway config is the product catalog. How the federation works Each domain is one AgentgatewayBackend that fans a single MCP endpoint across several real servers. A client connects once and sees the union of the targets’ tools: apiVersion: agentgateway.dev/v1alpha1 kind: AgentgatewayBackend metadata: name: mcp-billing namespace: mcp-federation spec: mcp: failureMode: FailOpen prefixMode: Always targets: - name: payments static: host: mcp-payments.mcp-federation.svc.cluster.local port: 80 path: /mcp protocol: StreamableHTTP - name: invoicing static: host: mcp-invoicing.mcp-federation.svc.cluster.local port: 80 path: /mcp protocol: StreamableHTTP Two settings here matter more than they look: prefixMode: Always exposes federated tools as <target>_<tool>, so payments_create_charge and invoicing_void_invoice are stable, predictable names. The authorization rules depend on that. Static targets, not selectors. A selector: target derives its tool prefix from the discovered Service name (mcp-payments-8080_get_payment), while a static target keeps the name: you declared. When policy matches on tool names, predictable naming is worth the extra three lines. failureMode: FailOpen keeps the federation serving healthy targets when one is down; the default, FailClosed, fails the whole session. How three identity providers share one gateway Each company has its own Keycloak realm, which stands in for each customer bringing their own IdP. One policy lists all three issuers; the gateway reads iss from the presented token, picks the matching provider, and verifies against that provider’s JWKS: spec: targetRefs: - group: gateway.networking.k8s.io kind: Gateway name: mcp-federation-gateway traffic: jwtAuthentication: mode: Strict providers: - issuer: http://keycloak.mcp-federation.svc.cluster.local:8180/realms/acme audiences: ["mcp-federation"] jwks: remote: url: http://keycloak.mcp-federation.svc.cluster.local:8180/realms/acme/protocol/openid-connect/certs cacheDuration: 5m # ... globex and initech providers, same shape mode: Strict closes anonymous access entirely, and audiences pins tokens to this gateway, so a token Acme minted for some other relying party is rejected even with a valid signature. Each token carries company and tier claims, and those two values drive everything that follows. Onboarding a fourth partner is four lines in this file plus an authorization rule; the repo’s scripts/add-partner.sh does it in one command. How per-customer tool entitlements work Authorization is CEL expressions on the backend, OR’d together, denied by default: backend: mcp: authorization: action: Allow policy: matchExpressions: - 'jwt.company == "acme" && mcp.tool.target == "payments" && mcp.tool.name in ["get_payment", "list_payment_methods", "create_charge", "refund_payment"]' - 'jwt.company == "globex" && mcp.tool.target == "payments" && mcp.tool.name in ["get_payment", "list_payment_methods"]' Initech has no billing entitlement, and its absence from this file is the entitlement. Initech authenticates successfully, reaches /mcp/billing, and sees an empty tool list. You grant access by adding a line, never by remembering to deny one. Two details I verified against the running gateway rather than assuming, because both are easy to get wrong: mcp.tool.name is the origin tool name (get_payment), not the federated name the client sees (payments_get_payment). The prefix is presentation; policy matches the real name. mcp.tool.target is required for correctness, not just tidiness. Without it, a rule allowing get_invoice also allows a same-named tool on any other server in the federation. Enforcement covers both directions. tools/list is filtered to what the caller may use, and a tools/call for anything else returns Unknown tool, so a caller can’t even confirm that an unentitled tool exists. The repo makes this visible in one command: scripts/mcp.py matrix That prints every tool, every company, side by side. My favorite detail: payments_void_transaction and invoicing_void_invoice are implemented, deployed, and running, and no company can reach them. The gateway is the only thing between “the capability exists” and “someone can invoke it”. Quotas that hold across gateway replicas Each company gets its own rate limit keyed on the same jwt.company claim the entitlements use: apiVersion: ratelimit.solo.io/v1alpha1 kind: RateLimitConfig metadata: name: company-quotas namespace: mcp-federation spec: raw: descriptors: - key: company value: acme rateLimit: requestsPerUnit: 600 # enterprise tier unit: MINUTE - key: company value: initech rateLimit: requestsPerUnit: 20 # trial tier unit: MINUTE # Catch-all: any company WITHOUT an explicit row gets its own counter - key: company rateLimit: requestsPerUnit: 30 unit: MINUTE rateLimits: - actions: - cel: expression: 'jwt.company' key: company The counters live in the Redis that ships with the enterprise install, so a limit holds across every gateway replica. 600 per minute means 600 total, not 600 per replica, which is the difference between a quota you can put in a contract and one that silently multiplies with your deployment size. The catch-all row is the part I’d steal for any multi-tenant config: a partner added tomorrow is rate-limited from their first request, under their own counter, without anyone editing this file. The safe default is automatic and an explicit row only ever grants more. Chargeback from gateway metrics agentgateway already counts MCP traffic by server, tool, and route. What the metrics lack by default is who called. One policy lifts claims off the validated JWT onto the request metrics: frontend: metrics: attributes: add: - name: company expression: jwt.company - name: tier expression: jwt.tier After that, chargeback is a single PromQL query: sum by (company, server, resource) (agentgateway_mcp_requests_total{method="tools/call"}) scripts/chargeback.py runs it, prices it against a rate card in scripts/pricing.json, and reports allowed versus denied versus throttled per company. --csv emits the same data for a billing pipeline. One cardinality warning from the manifest comments worth repeating: meter on the billable entity (company, tier), never on jwt.sub, or your time series count multiplies by your user count. I covered the LLM-spend version of this same pattern in Capping LLM Spend at the AI Gateway. Run it yourself Prerequisites: kubectl, helm, python3, k3d (or an existing cluster), and a Solo agentgateway license key. Validated against Solo Enterprise agentgateway v2026.8.0, Gateway API v1.5.0, Keycloak 26.0, and MCP protocol 2025-06-18. git clone https://github.com/themsquared/agentgateway-federate-mcp-example cd agentgateway-federate-mcp-example cp .env.example .env # add your AGENTGATEWAY_LICENSE_KEY ./setup.sh # creates a k3d cluster and installs everything In a second terminal: ./port-forward.sh # gateway :8080, Keycloak :8180, Prometheus :9090, UI :9080 Then either run the guided walkthrough (./demo.sh) or poke at it directly: scripts/mcp.py list acme billing # what one company sees scripts/mcp.py token globex # a token and its claims scripts/mcp.py quota initech 25 # watch the trial quota engage scripts/chargeback.py --by-tool # usage and cost by customer The six MCP servers are one stdlib-only Python file on a stock python:3.12-alpine image, with tools defined in ConfigMap JSON. No image builds, no registry. Edit the JSON, re-apply, done. Gotchas Things that cost me time, so they don’t cost you any: Keycloak’s issuer must be pinned. KC_HOSTNAME is set to the in-cluster service URL so tokens minted through kubectl port-forward still carry the in-cluster iss. Without it, a token fetched from your laptop claims iss=http://localhost:8180 and the gateway rejects it. Policies propagate through xDS. Give it a few seconds after kubectl apply before testing, or a just-applied entitlement looks broken when it’s just not there yet. Namespace discovery can strand your Gateway. If agentgateway was installed with discoveryNamespaceSelectors (common on a shared demo cluster), the controller ignores non-matching namespaces and the Gateway sits at Waiting for controller with no obvious cause. setup.sh detects this and labels the namespace automatically. Quota units are HTTP requests, not tool calls. One MCP session spends several (initialize, tools/list, one per call), so trial-sized quotas trip faster than the raw number suggests. Denied calls still count in agentgateway_mcp_requests_total, which has no status label. chargeback.py cross-references agentgateway_requests_total to separate allowed from denied and throttled; a production rate card should bill on successes only. Where this lands The pattern is simple to state: federate MCP servers by business domain, authenticate every caller against their own IdP, express entitlements as CEL in git, and let the same JWT claim drive authorization, quotas, and chargeback. The six servers never learn any of it, which means the next six servers won’t either. That’s the point of putting an AI gateway in front of MCP instead of teaching every server about every customer. The repo is themsquared/agentgateway-federate-mcp-example, including a WALKTHROUGH that builds it one layer at a time, an ONBOARDING guide for adding a partner with their own IdP, and a PRODUCTION doc mapping the POC to a real estate (50 IdPs, opaque tokens, per-user quotas). Next up, I want to wire agent-facing identity into the same federation: token exchange so the gateway swaps a caller’s JWT for scoped upstream credentials, which is where the agentic mesh story gets interesting. --- ## Thousands of AI Agents on Tens of Pods: kagent Agent Substrate - URL: https://webofmike.com/kagent-agent-substrate/ - Markdown source: https://webofmike.com/kagent-agent-substrate/index.md - Author: Mike Moore - Published: 2026-08-27 - Last modified: 2026-08-27 - Tags: Generative AI, Kubernetes, Tutorials, Platform Engineering - Description: kagent Agent Substrate is built to run thousands of AI agents on tens of pods. I stood it up on kind: snapshot restores, autoscaling, and every gotcha I hit. - Word Count: 1858 Agent Substrate is a Kubernetes-native runtime that breaks the pod-per-agent model. Instead of one always-on pod per AI agent, idle agents are checkpointed to object storage as compressed snapshots and restored on demand into a small pool of pre-warmed gVisor sandboxes. Agent count stops being a pod count: your fleet lives in object storage at near-zero marginal cost, and the design point is thousands of agents multiplexed across tens of pods. The kagent UI’s own source comments size its substrate inventory page for “a cluster running four hundred thousand” actors, which tells you where the project thinks this goes. I proved the mechanics end to end on a laptop: a kind cluster with kagent, nine agents on two worker pods, over a thousand real chat sessions served, reserved capacity 4.5x lower than the same agents as always-on pods, and a live visualizer that shows every restore, queue wait, and checkpoint as it happens. The whole thing, visualizer included, is in themsquared/kagent-substrate-demo on GitHub. The ratio is the same machine at any scale; only the numbers get bigger. What Agent Substrate is Always-on agent pods waste capacity because agents are idle most of the time. On my cluster, each of kagent’s default always-on agents holds about 204 MiB of memory and requests 50m CPU while doing nothing. Nine of those is roughly 1.8 GiB resident and 450m CPU reserved around the clock. Agent Substrate decouples the agent’s lifecycle from pod infrastructure: When an agent is invoked, its actor is restored onto a free worker from a WorkerPool, rehydrated from a zstd snapshot in object storage. The agent runs inside a gVisor sandbox for the duration of the session. One actor per sandbox, always. When the session ends, the actor’s state is checkpointed back to object storage and the worker slot frees for the next agent. A worker hosts one actor at a time. The density comes from time-multiplexing: worker-0 in my cluster served 50+ sessions across nine different agents in an afternoon. Simultaneous sessions equal your worker count; total agents are limited only by object storage. The runtime is two CRDs (WorkerPool and ActorTemplate in the ate.dev group) plus a control plane (ateapi, backed by a Valkey cluster), a data-plane router (atenet), a per-node snapshot mover (atelet), and a worker supervisor (ateom) that talks to gVisor’s runsc. A detail I enjoyed: atenet is agentgateway under the hood. Its config lives at /etc/agentgateway/config.yaml inside the pod. What the visualizer shows kubectl get pods makes substrate look boring because the interesting state is not in pods. The visualizer, Substrate Scope, renders the lifecycle directly: worker bays across the top, a restore queue below them, object storage at the bottom, and agent chips that physically move between the three as the cluster works. Below that, four telemetry charts compare reserved capacity against a pod-per-agent baseline measured from the real always-on agents in the same cluster. It has two modes. Simulated mode runs in any browser with no cluster, which is useful for talks. Live mode polls the kagent controller and renders your actual cluster, and its buttons are real: workers +/- runs kubectl scale workerpool, SURGE fires one genuine chat at every agent, and AUTOSCALE turns on a demand-based scaler that resizes the real WorkerPool. Standing it up on kind Prerequisites: kind, kubectl, helm, Docker, and either Ollama running locally or an LLM API key. Every command below ran successfully on my machine (Apple Silicon, Docker Desktop). Install Agent Substrate: kind create cluster --name kagent-substrate helm upgrade --install substrate-crds \ oci://ghcr.io/kagent-dev/substrate/helm/substrate-crds \ --version 0.0.6 --namespace ate-system --create-namespace --wait helm upgrade --install substrate \ oci://ghcr.io/kagent-dev/substrate/helm/substrate \ --version 0.0.6 --namespace ate-system --wait --timeout 10m Install kagent with the substrate integration. Two flags here are load-bearing and neither is in the official walkthrough: registry=ghcr.io (see the first gotcha below) and the Ollama provider, which needs no API key because the chart’s default host, host.docker.internal:11434, is exactly where kind on Docker Desktop finds your local Ollama: helm upgrade --install kagent-crds \ oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \ --version 0.9.9 --namespace kagent --create-namespace --wait helm upgrade --install kagent \ oci://ghcr.io/kagent-dev/kagent/helm/kagent \ --version 0.9.9 --namespace kagent --timeout 10m --wait \ --set registry=ghcr.io \ --set providers.default=ollama \ --set providers.ollama.model=qwen3:4b \ --set controller.substrate.enabled=true \ --set controller.substrate.ateApiEndpoint=dns:///api.ate-system.svc:443 \ --set controller.substrate.ateApiInsecure=true \ --set substrateWorkerPool.create=true \ --set substrateWorkerPool.replicas=2 \ --set substrateWorkerPool.ateomImage=ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.6 Deploy an agent onto substrate. A SandboxAgent carries the same spec as a regular kagent Agent, plus a substrate section. Note platform: substrate, which the docs example omits and the API requires: apiVersion: kagent.dev/v1alpha2 kind: SandboxAgent metadata: name: sre-oncall namespace: kagent spec: type: Declarative platform: substrate description: An SRE on-call assistant running as a substrate actor declarative: runtime: go modelConfig: default-model-config systemMessage: | You are an SRE on-call assistant. You triage alerts calmly. Keep answers to one or two sentences. substrate: workerPoolRef: name: kagent-default The first reconcile bakes a golden snapshot, which takes about a minute, and then the agent is a stored snapshot that costs nothing until someone talks to it. The repo has a nine-agent fleet manifest, and the visualizer with a load generator: node viz/server.mjs --live # visualizer at http://localhost:8123 node viz/stimulate.mjs --budget 300 # real chats until the budget runs out The --budget flag exists because I left the load generator running against the Anthropic API overnight and found out the hard way that it did not stop when I did. Where the per-actor state lives Kubernetes only sees WorkerPool and ActorTemplate. The state that makes the visualization interesting, which actor is running on which worker right now, lives in ateapi and reaches the outside world through a kagent controller endpoint: kubectl port-forward -n kagent svc/kagent-controller 8083:8083 curl -s http://127.0.0.1:8083/api/substrate/status The response carries four lists: workerPools, actorTemplates, actors (with live status like Resuming or Suspended), and workers (one row per worker pod, naming the actor it currently hosts). This is the same data the kagent UI’s Substrate page shows. One lesson that cost me an evening: derive assignments from the workers list, not the actors list. Every completed session leaves a Suspended actor entry behind, so the actors list grows without bound (mine passed 900 entries in a day) and starts returning partial results under load. The workers list is never larger than your pool and it names the assigned agent directly. When my visualizer joined through the actors list, running sessions randomly disappeared from the board. When I switched to the workers list, they stopped disappearing. What signal should autoscale a WorkerPool CPU is the wrong signal for this runtime, and it is worth understanding why before someone wires up a standard HPA. Workers are slot-bound: one actor per worker regardless of load, and an LLM turn is mostly I/O wait on the model provider. I measured workers hosting active sessions at single-digit mCPU. A CPU-based autoscaler would sleep through total saturation and scale down under peak load. What actually works is demand versus capacity: Scale up on queue depth. Substrate rejects new sessions when the pool is full rather than queuing them, so sustained rejections are the purest statement of unmet demand. Today that signal only exists client-side, in your retry loop. Scale down on the peak demand over a trailing window, not instantaneous demand, so a brief lull between sessions cannot slash the pool. Go straight to the target in one jump. busy + queued is the number of workers you need. Stepping by one with long cooldowns just makes users wait through several cycles. The autoscaler in the repo implements exactly that against the real CR, using the documented scaling path: kubectl scale workerpools.ate.dev kagent-default -n kagent --replicas=4 One side effect to know about: kubectl scale takes field ownership of .spec.replicas, so later helm upgrade runs on the kagent chart will fail with a server-side apply conflict until you add --force-conflicts. Gotchas Everything in this section happened to me on kagent v0.9.9 and substrate v0.0.6, with the verbatim errors you would search for. Agents stuck in Resuming forever My first SandboxAgent sat in Resuming for 25 minutes. The generated ActorTemplate pins the declarative runtime image by digest, cr.kagent.dev/kagent-dev/kagent/golang-adk@sha256:e014..., and that digest no longer exists in that registry: no such manifest: cr.kagent.dev/kagent-dev/kagent/golang-adk@sha256:e01479... Substrate retries the pull forever and the agent never becomes Ready. The identical digest still exists on ghcr.io, and every kagent image is mirrored there, so the fix is one flag at install time: --set registry=ghcr.io. After that change, all nine golden snapshots baked in about a minute each. spec.substrate may only be set when spec.platform is substrate The docs example for SandboxAgent fails validation on 0.9.9: The SandboxAgent "hello-substrate" is invalid: spec: Invalid value: spec.substrate may only be set when spec.platform is substrate Add platform: substrate to the spec. The field defaults to agent-sandbox. SandboxAgents are not on the regular A2A endpoint POST /api/a2a/kagent/<agent> returns Agent kagent/<agent> not found for SandboxAgents even when they are Ready. They live on a separate mount, and message/send requires a contextId: curl -s -X POST http://127.0.0.1:8083/api/a2a-sandboxes/kagent/sre-oncall/ \ -H 'Content-Type: application/json' \ -d '{"jsonrpc":"2.0","id":"1","method":"message/send","params":{"message":{ "kind":"message","messageId":"m1","contextId":"session-1","role":"user", "parts":[{"kind":"text","text":"Reply with the single word: ack"}]}}}' Without contextId you get message contextId (session id) is required for substrate sandbox agents. Worker pool has no free workers, but the pool looks idle A session that gets killed mid-flight (a timeout, a cancelled chat, an API outage) can leave its actor pinning a worker slot indefinitely, sometimes as a ghost that no longer appears in the actor inventory at all. New sessions then fail with substrate worker pool has no free workers while the pool looks empty. The fix is cheap because snapshots live in object storage, not in the pods: kubectl rollout restart deploy/kagent-default-deployment -n kagent Only the wedged sessions die. Every agent’s snapshot survives. In-sandbox egress fails until you restart substrate’s DNS Actor egress resolves through substrate’s own CoreDNS deployment, and mine wedged with plugin/reload: Corefile changed but reload failed. Agents could not reach Ollama on the host until I restarted it: kubectl rollout restart deploy/dns -n ate-system Changing the model requires re-baking the fleet Golden snapshots freeze the resolved model config at bake time. I switched the ModelConfig from Ollama to claude-haiku-4-5 and an existing agent kept answering from qwen. Delete and re-apply the SandboxAgents after any provider or model change; each re-bake takes about a minute and they queue through the pool. The model switch was worth it for demos, for what it’s worth. Sessions went from 30 to 120 seconds on a local 4B model to 1.5 to 2 seconds end to end, restore included. Wrapping up Nine agents on two pods, real snapshot restores you can watch, and an autoscaler driven by the signal that actually reflects demand. The code, the fleet manifests, the visualizer, and the load generator are all in themsquared/kagent-substrate-demo. The deeper runtime internals are documented at learn.agentsubstrate.dev and the substrate repo. Next on my list: wiring the queue-rejection signal into a proper KEDA scaler instead of my polling loop, and finding out whether substrate garbage-collects that ever-growing actor inventory. If you’re experimenting with agent runtimes on Kubernetes, this is a fun afternoon. I’ve been building agent tooling since the LLM agent playground, and this is the first runtime where the density story felt real on my own hardware. --- ## Revenant: A Gateway-Native, Always-On Agent Runtime - URL: https://webofmike.com/revenant-agent-runtime/ - Markdown source: https://webofmike.com/revenant-agent-runtime/index.md - Author: Mike Moore - Published: 2026-08-27 - Last modified: 2026-08-27 - Tags: AI Agents, Generative AI, Platform Engineering, Code - Description: Revenant is an always-on agent runtime in Rust where an AI gateway owns every key, budget, and failover, and a hard liveness rule means no turn goes silent. - Word Count: 1920 Always-on personal agents keep dying the same three deaths: leaked keys, runaway spend, and silence. The first two get the headlines. The third one is the one that actually made me want to throw my phone: you ask your agent to do something, it starts, and you never hear from it again. Revenant is my answer to all three. It is a lean agent runtime in Rust with chat channels, skills, cron loops, subagents, and graph memory, built on one architectural rule and one behavioral rule. The architectural rule: the harness never talks to a model provider directly. The behavioral rule: a turn always ends in a visible outcome, no exceptions. This post covers the concept, the model-tier design that makes a free local model the default with cloud failover behind it, and the night I found out the behavioral rule takes real engineering to keep. The code is public at themsquared/revenant. Why the harness never holds a provider key Every LLM, MCP, and A2A call Revenant makes flows through a bundled, supervised agentgateway. The gateway owns the provider keys, the model aliasing, cross-provider failover, token and dollar budgets, and the GenAI telemetry. The harness renders the gateway’s config; the gateway enforces it. you ──▶ telegram / web / tui / cli ──▶ revenant ──▶ agentgateway ──▶ any model, any MCP server (harness) (data plane: aliases, failover, budgets, guardrails, metrics) This split is the whole security story. Prompt injection cannot exfiltrate keys from a process that has never seen them. A runaway loop cannot blow past a spend cap that is enforced below the agent, in a data plane the agent has no ability to reconfigure. The agent gets a loopback URL and the names of model tiers. That is all it gets. It also means every hard reliability problem lives in exactly one place. Failover, health eviction, request logging, rate limits: these are gateway policies rendered from the harness config, not logic scattered through the agent. When I write about governing agent traffic through a gateway in other contexts, like an LLM-driven robot fleet, it is the same argument. Revenant just commits to it completely: there is no code path around the gateway to audit, because there is no code path around the gateway. How a free local model became the default Model access in Revenant is organized as tiers: fast, balanced, deep, and local. Each tier is a list of targets in priority order, and the gateway renders the list as one virtual model with failover routing. The agent asks for “balanced” and never learns which provider answered. When NVIDIA shipped Nemotron 3.5 Lightning on Ollama, the economics of the tier list flipped. It is a 30B mixture-of-experts model with 3B active parameters, built explicitly for the execution layer of always-on agents, and it runs comfortably on an M-series Mac. So my default tier now looks like this: [tiers.balanced] strategy = "failover" [[tiers.balanced.targets]] provider = "ollama" model = "nemotron-3.5-lightning:latest" reasoning_effort = "none" [[tiers.balanced.targets]] provider = "openai" model = "kimi-k3" api_key_env = "MOONSHOT_API_KEY" base_url = "https://api.moonshot.ai/v1" [[tiers.balanced.targets]] provider = "anthropic" model = "claude-sonnet-5" api_key_env = "ANTHROPIC_API_KEY" Priority 0 is a local model that costs nothing per token. The cloud models are failover, not the default. The gateway gives each failover member an outlier-detection policy, so a target that answers with “I am broken” status codes gets evicted for 60 seconds and the virtual model routes to the next priority: unhealthyExpression: response.code >= 500 || response.code == 529 || response.code == 429 || response.code == 402 || response.code == 401 || response.code == 403 || response.code == 404 eviction: duration: 60s I tested the direction everyone worries about: what happens when the local model is down. The first request fails with a 503 and evicts the Ollama target; every request in the eviction window cleanly serves from the cloud. The harness absorbs even that first failure, because the turn loop retries once when nothing has streamed yet. Ollama dies, the user notices nothing, and the bill goes up by one cloud call. The reverse direction matters too: when a cloud account runs out of credit, the same eviction machinery routes back down. Deep reasoning stays on a cloud tier by choice, not by dependency. Local reasoning models break agents in two specific ways Nemotron is a reasoning model, and putting a reasoning model at priority 0 of an agent runtime surfaced two failure modes worth knowing before you try this. First, Ollama rejects forced tool selection while thinking is enabled. Any harness feature that pins tool_choice to a specific tool for structured output gets this back: {"type":"error","error":{"type":"invalid_request_error","message":"tool_choice 'specified' is incompatible with thinking enabled"}} In Revenant that broke every structured-output path at once, including the agent’s own self-review loop, which retried on a five-minute cadence and failed identically every time. Second, reasoning is silent. The thinking tokens do not translate into streamable output through the gateway’s Anthropic-shape endpoint, so the user watches nothing happen for minutes. And when the reasoning burn exceeds the request’s max_tokens, the final message arrives empty. From the outside, both look like a dead agent. The fix for both was one knob. Ollama’s OpenAI-compatible endpoint accepts reasoning_effort: "none", and agentgateway can force request-body values per model with a static override. Revenant now renders that from the reasoning_effort line in the tier config above: - name: balanced/0 params: model: nemotron-3.5-lightning:latest overrides: reasoning_effort: none provider: ollama With thinking off, forced tool calls return clean tool_use blocks and answers stream immediately. The local tier’s job is fast execution. Deliberate thinking belongs to the deep tier, where a frontier model does it better anyway. A turn can never go silent Here is the behavioral rule, stated the way I gave it to the project: a conversation never goes without a response, and a job never fails to report. An agent that errors is annoying. An agent that goes quiet is useless, because you cannot tell it apart from an agent that is working. I thought the rule held until I went looking. One debugging night turned up three independent holes, and each one alone produces a silent turn. Unbounded waits. The LLM client had a connect timeout and nothing else, on the theory that streaming responses can be long-lived. True, and also fatal: a stalled stream (half-open TCP, wedged upstream) hung the turn forever. Worse, the session model treats messages that arrive during a running turn as mid-turn interjections, so every follow-up you send while a turn is hung gets silently queued behind a turn that will never finish. The fix is layered deadlines on every stream: time to first byte, maximum idle gap between events, and a total per-attempt ceiling, each tunable by environment variable. A stall becomes an error, and an error becomes a reply. No backstop. Per-step deadlines cannot cover what they cannot see, so the session actor now races every turn against a wall-clock watchdog. On expiry it cancels the turn, grants a grace period to unwind, then drops the future outright and reports the overrun. The drop is safe because the turn’s session state lives in an RAII guard that unwinds on any exit path. Restart amnesia. This was the one actually firing, and it had nothing to do with models. A turn in flight when the daemon stops just dies with the process, and nothing ever told the owner. My canary box made it comic: the auto-rollout script compared the full git SHA against a marker file it had written in short form, the comparison could never match, and so the deploy timer concluded a new deploy was needed every five minutes. Every turn longer than the window died mid-flight, all day, every day. That single character class, short SHA versus full SHA, was the original “my agent never finishes anything” bug. The rollout comparison is a one-line fix. The structural fix is that the daemon now sweeps on boot for sessions whose most recent message is the user’s, which is precisely a turn that was accepted and never answered, and reports each one: “the daemon restarted while this turn was running, the task did not complete.” The notice is persisted into the session transcript, so the next boot does not re-report it and the conversation keeps an honest record of where the turn was cut. Validation is a soak harness that speaks to the live control API: submit a turn, then require a terminal event within a deadline, with verdicts of OK, SLOW, or SILENT. SILENT is an invariant violation, full stop. The final run on a stable daemon: [1/6] OK turn_completed 33s 'What is 41*17? Reply with just the number.' [2/6] OK turn_completed 43s 'Say hello in exactly three words.' [3/6] OK assistant_reply 121s 'What time is it right now? Use a tool if you have one.' [4/6] OK turn_completed 108s 'List two of your available skills, one line each.' [5/6] OK turn_completed 67s 'What is 12! (twelve factorial)? Just the number.' [6/6] OK turn_completed 94s 'Name your current model tier in one sentence.' PASS: invariant held on all 6 turns (0 slow, 6 within SLA) The run before that one is the more interesting result: a mid-soak deploy killed a turn on purpose, and the sweep on the next boot converted it into a reported failure instead of silence. The invariant held through the exact event that used to break it. The agent cut its own release One more thing happened that night that I want on the record because it is the point of the project. After the liveness fixes merged, I went to tag the release and found the tag already existed. Revenant’s self-improvement loop had watched its changes land on main, tagged v2026.8.0 itself, and handed the tag to CI, which built and published the release. The tagger on the annotated tag is the agent’s own identity. That loop, called ascension, runs the same way everything else does: through the gateway, under budgets, behind an approval broker with default-deny, gated by evals, and with a PR-based path so a human review sits between the agent and its own code. Self-releasing sounds alarming until you notice the agent has strictly fewer privileges than the CI system most teams already trust. A runtime where the agent maintains itself is exactly where always-on agents are going, and I would rather learn the failure modes on my own fleet of two machines than read about them later. If the scale end of this interests you, I wrote about running thousands of agents on tens of pods with kagent’s substrate. Try it The installer fetches pinned, checksum-verified binaries for the harness and gateway, then walks through provider setup: curl -fsSL https://raw.githubusercontent.com/themsquared/revenant/main/installer/install.sh | sh revenant chat # supervised gateway + streaming REPL revenant service install # run it always-on (launchd / systemd) revenant open # web UI: chat, approvals, spend, loops No cloud key is required. With Ollama installed, a local tier serves everything, and the tier config above is the template for adding cloud failover later. Where this goes Revenant is my working answer to what an always-on personal agent should be: a harness that cannot leak what it does not hold, a gateway that enforces what the agent cannot override, a free local model doing the everyday work with the cloud as backup, and a liveness rule with teeth. The next steps are making local-first the setup wizard’s default when Ollama is present, and letting the two-machine fleet keep upgrading itself. The code, the lore, and the tier configs are all in themsquared/revenant. --- ## Governing an LLM-Driven Robot Fleet with agentgateway - URL: https://webofmike.com/llm-robot-fleet-agentgateway/ - Markdown source: https://webofmike.com/llm-robot-fleet-agentgateway/index.md - Author: Mike Moore - Published: 2026-08-27 - Last modified: 2026-08-27 - Tags: Generative AI, AI Agents, Robotics, Platform Engineering - Description: An LLM-driven robot fleet where every inference, MCP tool call, and agent message flows through agentgateway: safety, failover, and speed, all open source. - Word Count: 1918 I built a simulated warehouse robot fleet where LLMs do the planning and every piece of traffic that crosses the network runs through one open source gateway. Every inference request, every MCP tool call that moves a robot, and every agent-to-agent message goes through agentgateway, and the gateway’s config file is the complete, reviewable list of everything the fleet is allowed to do. The code is at github.com/themsquared/physical-ai-demo. It runs on Docker Compose with zero cloud keys, and the same images run on a Jetson. Why physical AI needs a governed connectivity layer LLM output is stochastic. Hardware is not forgiving about that. A chatbot that hallucinates gives you a bad paragraph; a robot arm that hallucinates gives you an incident report. Physical AI works only if the architecture converts probabilistic model output into bounded, auditable, deterministic-enough physical action. That conversion doesn’t happen inside the model, and it doesn’t happen in servo firmware. It happens in the layer between them: the connectivity layer. The demo is a working argument that this layer can be built entirely from open source today, with agentgateway at the center, and that five properties hardware demands are all enforceable there: Safety. The model may want anything; the machine may only do allowed things, and every action is attributable to an identity. Failover. Cognition will drop (WAN loss, model crash, GPU contention). The machine must degrade, not flail. Speed. Middleware overhead must be near zero and measured, and real-time decisions must never depend on the network. Repeatability. Identical behavior across robots, sites, and runs, checkable rather than anecdotal. Predictability. The envelope of possible actions must be statically knowable before deployment, whatever the model outputs. Each pillar has a runnable acceptance test in the repo (make verify-safety, make verify-failover, and so on), so the claims are checkable, not vibes. How the fleet is wired Three tiers, deliberately separated: Reflex tier: in-process control loops inside each robot (e-stop, human-in-zone refusal). Millisecond scale, never proxied, never on the network. This is the reason the machine stays safe even when every cable is cut. Cognition tier: one LLM planning loop per robot. Inference traffic. Coordination tier: robot-to-robot and orchestrator traffic over A2A. The gateway governs the two tiers that cross the network. The reflex tier is in-process on purpose; that placement is the Safety and Speed argument in one design decision. OPERATOR ──A2A──▶ agentgateway :3000/:4000 ──▶ OTel │ audit │ Prom ├── /llm failover chain + per-identity token budgets ├── /mcp/* deny-by-default CEL authz + JWT + audit └── /a2a/* agent-to-agent routing │ ┌───────────────────────────────┼────────────────────────────┐ orchestrator (A2A) cognition agents (A2A) ollama-primary mission → plan → delegate per robot: LLM loop, tools ollama-fallback ONLY via the gateway mock-llm (deterministic) │ MCP (via gateway) ┌───────────┴───────────┐ amr-1 / amr-2 arm-1 (MCP servers) + reflex tier (in-process, never on the network) + degraded-mode state machine │ warehouse-world (seeded sim) The robots are MCP servers (two autonomous mobile robots and one arm). The cognition agents are plain Python with a stock OpenAI client and a small hand-rolled A2A surface, no agent framework. That’s deliberate: the interesting behavior lives in the gateway config, not in framework glue. How agentgateway authorizes robot tool calls The whole fleet’s capability surface lives in one file, gateway/config.yaml. MCP authorization is deny-by-default: a tool call is refused unless a CEL rule allows it. Here is the actual policy for one robot: - name: mcp-amr-1 gateways: [default] matches: - path: pathPrefix: /mcp/amr-1 policies: mcpAuthorization: rules: # A robot's own cognition may use its tools EXCEPT the gated three. - 'jwt.sub == "amr-1-cognition" && !(mcp.tool.name in ["disable_safety_stop", "set_torque_limit", "calibrate"])' # The orchestrator sees telemetry only — it delegates, it never actuates. - 'jwt.sub == "orchestrator" && mcp.tool.name in ["get_pose", "get_battery", "get_state"]' # Maintenance identity: full surface, including gated tools. - 'jwt.sub == "maintenance"' Three properties fall out of this: Gated tools are invisible, not just refused. agentgateway filters unauthorized tools out of tools/list, so a robot’s LLM never even sees disable_safety_stop in its tool schema. In the demo, an adversarial orchestrator prompt tries to disable a safety stop; the tool isn’t listed, the forced call is denied, and the denial lands in the audit log. Every action is attributable. Each agent carries its own JWT, and the gateway writes a JSON audit record for every tool call with the tool name, arguments, and identity. When a robot moves, you can answer “who asked for that, with what arguments, and under which policy” from one log. The action envelope is a file. Adding a capability to the fleet means changing this YAML in a pull request. The envelope is diffable with git diff and reviewable like any other code. That is the Predictability pillar in practice: you don’t need to predict the model, you need to bound it. make verify-safety asserts all of it: 100% of gated tools invisible and denied for non-maintenance identities, 100% of actions audited with args and identity, and the e-stop path in-process at 10ms or less, never on the wire. How LLM failover keeps a robot from flailing Cognition loss is a when, not an if. The gateway exposes one virtual model to every agent and runs a priority failover chain behind it: virtualModels: - name: robot-brain # the only model name agents know routing: failover: targets: - model: primary priority: 0 - model: fallback priority: 1 - model: mock priority: 2 The primary and fallback rungs are Ollama models (qwen3:4b and qwen2.5:0.5b by default); the bottom rung is an in-repo deterministic OpenAI-compatible server called mock-llm. Each rung has health-based eviction, so a model that starts returning 400s (including the 404 you get when a model was never pulled) is evicted after one failure. Kill the primary Ollama container mid-mission and the next request is served by the fallback, visibly in the metrics. The chain always ends on mock-llm, which means the demo degrades to a working deterministic brain even on a machine with no models pulled at all. The same rung is what CI uses: no GPU, no keys, bit-exact. Failover alone isn’t enough, though. Each robot also runs a degraded-mode state machine (ACTIVE → SAFE_IDLE → RESUME): if a robot’s cognition dies entirely, the robot safe-idles within 500ms of simulated time, the rest of the fleet keeps working, and the orchestrator reassigns the mission. Token budgets get the same treatment. The gateway enforces a fleet-wide token budget, and exhaustion produces a clean 429 that the robot handles by safe-idling rather than retrying into a dead brain. What the gateway costs in latency Putting a proxy between a robot and its tools is only defensible if the cost is measured. agentgateway’s data plane is Rust, and the repo ships a bench harness (bench/bench.py) that measures the same MCP call direct versus through the gateway and asserts the SLO: p95 added overhead of 10ms or less, with around 4ms typical. The more important speed decision is architectural. Nothing real-time goes through the gateway at all. The reflex tier (collision refusal, e-stop, human-in-zone stops) is in-process inside the robot with zero network hops. The gateway governs planning and coordination, which tolerate milliseconds; the control loop tolerates nothing, so it never left the process. How repeatability gets checked without re-running missions Every mission emits OpenTelemetry traces through the gateway. The eval suite feeds those traces to agentevals, which scores a run’s tool-call trajectory against a golden trace without re-executing anything. Run the same seeded mission twice and make verify-repeat asserts zero tool-sequence drift across runs; in CI the same gate fails a pull request on behavioral regression. The pieces that make this possible: a seeded deterministic warehouse world, the deterministic mock-llm rung for bit-exact CI runs, and one declarative gateway config that is byte-identical between the laptop sim and the Jetson deployment. The chaos suite covers the adversarial side. Twenty-plus hostile prompts, including one injected through a pallet label that the LLM reads out of world state (the sim has a built-in prompt-injection vector on purpose), must produce zero envelope escapes. Escapes are judged from ground truth, meaning world state plus the audit log, never from what the model claims it did. Quickstart Docker and Docker Compose are the only requirements. Everything is open source and no cloud keys are involved. make setup # venv + demo JWT material (RSA keypair, per-identity tokens) make up # full stack (first run pulls Ollama models — see "Models" below) make demo # the five-act narrated demo (add AUTO=true to run hands-free) That gives you a live top-down warehouse view at http://localhost:8085/, a Grafana dashboard with one row per pillar at http://localhost:3001, and the agentgateway UI at http://localhost:15000/ui/. To send a mission yourself: bash scripts/run-mission.sh # sends the default mission over A2A And to check every pillar’s acceptance test in one shot: make verify-all Gotchas Docker on a Mac gets no GPU. Containerized Ollama runs on CPU, which makes a live demo sluggish. Point the primary rung at host Ollama instead, which is Metal-accelerated: ollama pull qwen3:4b # on the host echo 'PRIMARY_BASE_URL=http://host.docker.internal:11434/v1' >> .env Stale backend DNS after a rebuild. If you rebuild a backend image while the gateway is running, the gateway can hold a resolved IP for a container that no longer exists. The Makefile starts the gateway last and force-recreates it on make up, and there’s a dedicated target for the mid-session case: make reset-gateway # re-resolve backends (run after rebuilding any backend image) Token budgets need request-time tokenization. A budget that’s only debited from provider usage after the response can’t reject a request up front. Every model rung in the config sets tokenize: true so the gateway estimates tokens at request time and budget exhaustion produces the 429 before the inference happens. Small local models flub tool calling. A 4B model will sometimes emit malformed tool calls or claim success it didn’t have. The demo doesn’t pretend otherwise. Agent output is schema-constrained with bounded retries, the failover chain ends on a deterministic rung, and the chaos suite judges from world state and the audit log rather than from model output. Kubernetes and the Jetson The compose stack is the demo path, but k8s/ carries a Kustomize base with k3d and Jetson overlays. make k8s-up stands the whole thing up on k3d, and a kagent fleet-sre agent runs as the ops tier: make verify-kagent has it diagnose and remediate a crashlooping robot. All images are multi-arch, so the same manifests deploy to a Jetson Orin Nano with qwen3:1.7b on the primary rung. The hardware punchline is that the pillar table doesn’t change. The robot code is written against a Driver interface; swapping the simulated arm for a LeRobot SO-101 means writing a LeRobotDriver while the gateway YAML, the CEL envelope, the audit log, and the evals all stay identical. Only the actuator changes. Physical e-stop wiring stays hardware-side, never on the network path. What’s next The demo proves the connectivity-layer argument on a simulated fleet: deny-by-default tool authorization, LLM failover with graceful degradation, measured single-digit-millisecond overhead, trace-based repeatability scoring, and an action envelope you can review in a pull request, all on an open source stack. The code, the config, and every verify target are at github.com/themsquared/physical-ai-demo. Next up is the hardware track: a real SO-101 arm behind the same gateway config, and eventually a physics-accurate sim (ROS 2 and Gazebo) to replace the kinematic world. I wrote more broadly about where this space is heading in The Future of AI and Robotics; this project is what the infrastructure side of that future looks like when you build it. --- ## Capping LLM Spend at the AI Gateway: Budgets and Virtual Keys - URL: https://webofmike.com/llm-cost-controls-ai-gateway/ - Markdown source: https://webofmike.com/llm-cost-controls-ai-gateway/index.md - Author: Mike Moore - Published: 2026-08-27 - Last modified: 2026-08-27 - Tags: Generative AI, Kubernetes, AI Gateways, Platform Engineering - Description: Token counts are not costs. I wired a price catalog, virtual keys, and budgets into agentgateway so LLM spend gets attributed, priced, and capped with a 429. - Word Count: 1640 Every agent platform demo eventually meets the same question from the person who signs off on it: who is spending what, and what stops a runaway agent from spending more? I added that layer to my Solo agentic stack demo as an eighth act. It gives every LLM request a price, attributes it to a user and a team, and refuses the request with an HTTP 429 once a budget is gone. All of it lives at the gateway, which is the only place you can do it once for every agent, every framework, and every provider at the same time. The manifests are in manifests/cost-management/ and the walkthrough is ./demo.sh --act 8. Why token counts are not costs Gateways have counted tokens for a long time. Counting is not costing. Without prices, every spend number in your dashboard is zero, and “we used 4.1 billion tokens last month” is not a number anyone can act on. Three things have to line up before a dollar figure appears, and the important part is that each one fails quietly rather than loudly: Prices. A per-model price catalog attached to the Gateway. Attribution. A key that says which user and which team a request belongs to. Enforcement. A budget that reads the first two and blocks when exceeded. Miss the first and you get traffic charts with zero dollars. Miss the second and everything lands in a bucket called Unattributed. Neither throws an error. Loading a model price catalog The catalog is JSON keyed by provider, then model, with rates as exact decimal strings in USD per million tokens. Strings, not floats, so nothing gets rounded on the way in. apiVersion: v1 kind: ConfigMap metadata: name: agw-model-costs namespace: agentgateway-system data: catalog.json: | { "providers": { "anthropic": { "models": { "claude-sonnet-4-6": { "rates": { "input": "3.00", "output": "15.00", "cacheRead": "0.30" } } } }, "openai": { "models": { "gpt-4o": { "rates": { "input": "2.50", "output": "10.00", "cacheRead": "1.25" } } } } } } The Gateway picks it up through modelCatalog.sources on the EnterpriseAgentgatewayParameters it already references: spec: modelCatalog: sources: - configMap: name: agw-model-costs key: catalog.json Order matters. Apply the ConfigMap before the parameters that point at it, or the catalog only takes effect on a later reconcile. Once loaded, the gateway populates a llm.cost field on every request that flows into access logs, traces, and metrics. Hand-maintaining prices is a bad idea long term. agctl costs import --providers openai,anthropic generates the catalog from shipped price data. Virtual keys, and how attribution actually resolves A virtual key is a gateway-issued token carrying attribution metadata. The caller sends one key, the gateway resolves it into cost dimensions, and the real provider credential never leaves the gateway. apiVersion: v1 kind: Secret metadata: name: llm-virtual-keys namespace: agentgateway-system labels: agentgateway.solo.io/virtual-key-set: demo type: Opaque stringData: alice: | { "key": "sk-alice-demo", "metadata": { "id": "alice", "user": "alice", "group": "platform-eng" } } The mapping from that metadata to budget dimensions is worth writing down, because it is not obvious from the resource: virtualKey resolves from apiKey.id, which is metadata.id above. user resolves from a coalesce over apiKey.user, apiKey.name, apiKey.owner, jwt.sub, jwt.email and more. group resolves from coalesce(jwt.group, apiKey.group). model and provider are built in and need no expression. Those defaults live in the chart’s budgetDimensions.config. Custom dimensions (cost centre, tenant) are CEL expressions added to the same block. The policy that consumes the key set uses mode: Strict, which is a deliberate choice. Strict means a valid key is required, so on this route unattributed spend cannot happen at all rather than merely being visible: traffic: apiKeyAuthentication: mode: Strict secretSelector: matchLabels: agentgateway.solo.io/virtual-key-set: demo entBudgetEnforcement: {} entBudgetEnforcement is the switch that makes budgets apply to this route. Without it the budgets exist and do nothing. Writing budgets in tokens and dollars One EnterpriseAgentgatewayBudget holds up to 64 independent entries. Each is a subject, a limit, a window, and an action. apiVersion: enterpriseagentgateway.solo.io/v1alpha1 kind: EnterpriseAgentgatewayBudget metadata: name: cost-demo-budgets namespace: agentgateway-system spec: budgets: - name: per-key-daily-tokens subject: virtualKey: "*" limit: unit: Tokens amount: 100 window: unit: Day onBudgetExceeded: Block - name: platform-eng-monthly-usd subject: group: platform-eng limit: unit: USD amount: 50 window: unit: Month onBudgetExceeded: Audit subject.virtualKey: "*" gives each key its own bucket, so one caller running dry has no effect on another. Block returns a 429. Audit records the breach and lets the request through, which is how you roll budgets out without causing an outage on day one. Windows are rolling from the first debit, not calendar aligned. USD limits are whole numbers, so one dollar is the smallest budget you can write. Dollar budgets also depend entirely on the catalog: a model the catalog cannot price adds zero, so an incomplete catalog undercounts silently. The controller confirms it took the resource: kubectl get eagbud cost-demo-budgets -n agentgateway-system \ -o jsonpath='{.status.conditions[*].reason}' # Valid Enforced Proving enforcement with a 429 This is the part worth showing live. Six requests succeed, the seventh does not, and a second key on its own bucket is untouched. alice request 1: HTTP 200 alice request 2: HTTP 200 alice request 3: HTTP 200 alice request 4: HTTP 200 alice request 5: HTTP 200 alice request 6: HTTP 200 alice request 7: HTTP 429 bob (own bucket): HTTP 200 A 100 token daily budget against gpt-4o with max_tokens: 20 works out to roughly 17 tokens per “hi” (8 in, 9 out), so the flip lands on the seventh request. My first version of the demo loop ran six requests and stopped, which showed six 200s and no 429, the exact opposite of the point. Count your own token cost and make the loop run past it. Checking that pricing actually landed The failure mode here is silence, so verify with the counter rather than trusting the dashboard: kubectl port-forward deploy/agentgateway-proxy -n agentgateway-system 15020:15020 & curl -s localhost:15020/metrics | grep agentgateway_cost_catalog_lookups_total agentgateway_cost_catalog_lookups_total{status="Exact",gen_ai_system="openai", gen_ai_request_model="gpt-4o",gen_ai_response_model="gpt-4o-2024-08-06", route="agentgateway-system/metered-llm"} 2 status="Exact" means the request was priced from the catalog. Missing, Unpriced, or NoCatalog mean spend is being undercounted, and nothing anywhere will tell you that except this counter. Two caveats worth saying before someone asks Both of these are properties of the design, and a FinOps buyer respects hearing them up front more than a claim of a hard cutoff. Enforcement is approximate. Token counts are not known until the response comes back, so usage is debited after the fact. A burst can overshoot the limit slightly before the next request is refused. Budgets fail open. If the rate limit service is unreachable, requests are allowed through. Availability is chosen over a hard spend cutoff. Why budgets never trip when a provider is down Budgets debit usage from the response. A provider that refuses the request produces no usage, so nothing is debited and the budget never trips. In my case one provider account was out of credit and the gateway faithfully passed the error back: {"error":{"type":"invalid_request_error", "message":"Your credit balance is too low to access the Anthropic API."}} The gateway did its job. Authentication passed, the request was priced, the upstream was called. But the whole cost-control act rendered as a row of 400s instead of the 429 it exists to show. I pointed the metered route at the provider that was actually answering, which also insulates the act from any single provider’s billing state. Four silent failures found while bumping versions Moving the stack to current releases (agentgateway v2026.8.2, kagent Enterprise 0.5.5, AgentRegistry 2026.8.0) surfaced a pattern: none of these failed loudly. Helm accepts unknown --set keys without complaint. kagent 0.5.x renamed the built-in agent toggle from agents.k8s-agent.enabled to k8s-agent.enabled. The old key produced no error, just a missing agent. helm template settles it in seconds: helm template k ./kagent-enterprise --set k8s-agent.enabled=true | grep -c "^kind: Agent" # 1 helm template k ./kagent-enterprise --set agents.k8s-agent.enabled=true | grep -c "^kind: Agent" # 0 A stale pin in .env silently overrides the script. My setup script uses ${VAR:-default} and sources .env first, so an old AGW_VERSION in a gitignored file quietly won every version bump I made to the script. Worth checking before concluding a bump did not work. kubectl wait does not wait for objects that do not exist yet. The mesh readiness check looked patient and was not: Error from server (NotFound): error when creating "istiod-alias.yaml": namespaces "istio-system" not found Against a namespace that has not been created, kubectl wait fails in milliseconds rather than blocking, so the advertised 300 second timeout was never spent. The script raced ahead into a fatal apply and died under set -e. The fix is to poll for existence first, then wait on conditions. A new required field breaks one object and not the rest. AgentRegistry 2026.7.x began requiring config.auth.oidc on kagent runtimes: ✗ Runtime/kagent failed: prepare: invalid input: auth.oidc is required for kagent runtimes ✓ MCPServer/weather-mcp (latest) created ✓ Agent/weather-assistant (latest) created Everything else registered fine. The catalog looks healthy right up to the moment you try to deploy an agent and discover there is no runtime to deploy it onto. Running it git clone https://github.com/themsquared/agentic-demo cd agentic-demo cp .env.example .env # add LLM keys and a Solo license ./setup.sh # cluster, mesh, gateway, kagent, registry ./port-forward.sh ./demo.sh --act 8 # cost management, live Act 8 applies the same files in manifests/cost-management/ that this post quotes, so the YAML on screen is the YAML that runs. What is next The dimensions model is the interesting thread. Cost centre and tenant are CEL expressions away, which makes chargeback per team a config change rather than a data pipeline. I am also curious whether budgets can key off an agent identity rather than a human one, since in an agentic mesh the caller is usually another agent. If you want the layer underneath this, I wrote up running thousands of agents on tens of pods with kagent Agent Substrate. The code for everything here is at themsquared/agentic-demo. --- ## Building an LLM Agent Playground: A Deep Dive into AI Orchestration and Evaluation - URL: https://webofmike.com/building-an-llm-agent-playground/ - Markdown source: https://webofmike.com/building-an-llm-agent-playground/index.md - Author: Mike Moore - Published: 2025-02-23 - Last modified: 2025-02-23 - Tags: Generative AI, AI Training, DevOps - Description: An open-source LLM Agent Playground for experimenting with, evaluating, and comparing different LLM providers through a unified interface. - Word Count: 686 Introduction In the rapidly evolving landscape of artificial intelligence and machine learning, Large Language Models (LLMs) have emerged as powerful tools for natural language processing and generation. However, the real potential of these models lies not just in their ability to understand and generate text, but in their capacity to act as intelligent agents that can perform concrete actions in response to natural language instructions. Inspired by the course at DeepAtlas.ai in AI orchestration and agent systems, an open-source LLM Agent Playground has been developed that allows developers and researchers to experiment with, evaluate, and compare different LLM providers through a unified interface. This project serves as both a practical tool and an educational resource for understanding how to build agentic systems with modern AI technologies. LLM Agent Playground GitHub Repository The Rise of Agentic AI Systems The concept of agentic AI systems represents a significant evolution in artificial intelligence. Unlike traditional LLMs that simply respond to prompts, AI agents can: Understand user intentions Plan sequences of actions Execute concrete tasks Learn from feedback Adapt to changing contexts This shift from passive language models to active agents marks a crucial step toward more practical and impactful AI applications. Key Features of the LLM Agent Playground Multi-Provider Support The playground integrates with multiple LLM providers: OpenAI’s GPT-3.5 and GPT-4 Anthropic’s Claude Local models through Ollama This multi-provider approach allows for comprehensive comparison and evaluation of different models’ capabilities and cost-effectiveness. Action System Architecture At the heart of the playground lies a flexible action system that transforms language models into capable agents. Each action is a well-defined capability that models can invoke, following a structured protocol: class CustomAction(BaseAction): name = "custom_action" description = "Performs a specific task" required_parameters = { "param1": "First parameter description", "param2": "Second parameter description" } This architecture enables: Structured output validation Clear parameter specifications Comprehensive error handling Automatic action discovery and registration Evaluation and Analytics The playground includes robust tools for: Comparing model performances Tracking response quality Monitoring costs Visualizing trends over time This data-driven approach helps organizations make informed decisions about which models best suit their specific needs. Building Blocks of an Agent System 1. Language Model Integration The system abstracts away the complexities of different LLM providers through a unified interface: Consistent API patterns Standardized response formats Unified error handling Cost tracking and optimization 2. Action Framework The action system follows clean design principles: Modular action definitions Automatic registration Clear parameter validation Structured error handling Comprehensive logging 3. Evaluation Infrastructure Built-in evaluation capabilities include: Response ranking Cost analysis Performance trending Export functionality Practical Applications 1. Model Evaluation Organizations can use the playground to: Compare model capabilities Assess cost-effectiveness Measure response quality Track performance trends 2. Prototype Development Developers can: Test new action implementations Experiment with different models Optimize prompts Validate user experiences 3. Research and Analysis Researchers can: Study model behaviors Collect performance metrics Analyze cost patterns Compare provider capabilities Technical Implementation Backend Architecture Python-based API server PostgreSQL database Async request handling Modular provider integration Frontend Design React-based UI Real-time updates Interactive visualizations Responsive design Action System Auto-discovery mechanism Structured validation Comprehensive logging Error handling Getting Started Prerequisites Python 3.11+ Node.js 16+ PostgreSQL 13+ Ollama for local models Basic Setup Clone the repository Set up the Python environment Configure the database Install required models Set up environment variables Start the application Future Directions The LLM Agent Playground opens up several exciting possibilities: Enhanced evaluation metrics Additional provider integrations More sophisticated action chains Improved visualization tools Advanced cost optimization Conclusion The LLM Agent Playground represents a significant step forward in making AI agents more accessible and practical. By providing a unified interface for working with multiple LLM providers and a robust action system, it enables developers, researchers, and organizations to build and evaluate agentic AI systems effectively. The project demonstrates how modern AI technologies can be orchestrated to create practical, actionable systems while maintaining transparency, cost-effectiveness, and performance optimization. Get Involved The project is open-source and welcomes contributions. Whether you’re interested in adding new features, improving documentation, or sharing your experiences, there are many ways to get involved. LLM Agent Playground GitHub Repository --- ## The Future of AI and Robotics: A Look at What's Next - URL: https://webofmike.com/the-future-of-ai-and-robotics/ - Markdown source: https://webofmike.com/the-future-of-ai-and-robotics/index.md - Author: Mike Moore - Published: 2025-02-15 - Last modified: 2025-02-15 - Tags: AI Consulting, DevOps - Description: Exploring the rapidly advancing world of AI and robotics, from autonomous vehicles to humanoid robots, and what it means for humanity. - Word Count: 904 The World is Changing Fast Imagine waking up in a world where robots perform surgeries, self-driving cars fill the highways, and AI assistants handle everything from booking your meetings to writing reports better than you do. That future? It’s already happening. Artificial Intelligence (AI) and Robotics are advancing at an unprecedented pace, transforming industries, reshaping economies, and challenging us to rethink what it means to work and live alongside machines. But with all this potential, we also face serious questions: Will AI take our jobs? How will robotics impact our daily lives? What risks do these technologies pose to society? Let’s dive into where we are, where we’re heading, and what it means for humanity. 1. The Present: Where Are We Now? AI and robotics are no longer confined to science fiction—they’re here, and they’re changing the world. AI in Everyday Life — AI is already deeply embedded in our lives. Chatbots answer customer service inquiries, AI-driven algorithms make stock market predictions, and machine learning helps doctors diagnose diseases with greater accuracy than ever before. Autonomous Vehicles & Robotics in Industries — Self-driving cars from companies like Tesla, Waymo, and Cruise are in development, and AI-powered robots are dominating warehouses, manufacturing plants, and even fast-food kitchens. The Rise of Humanoid Robots — Companies like Boston Dynamics, Tesla, and SoftBank are building robots that can walk, run, and even hold conversations. Some are already assisting in elderly care, security, and disaster response. The key takeaway? AI and robotics are no longer a futuristic fantasy—they’re an evolving reality. 2. The Future: Where Are We Headed? The next decade will bring even more radical transformations. Here are some of the biggest developments we can expect: AI and Automation Revolution Robots and AI will take on more complex roles, from cooking food to managing entire warehouses. By 2030, up to 45% of jobs could be automated, according to some estimates. However, this won’t be a simple replacement—it will be a shift, requiring new skills and new jobs to emerge. Human-Robot Collaboration Rather than replacing humans, robots will work alongside us, enhancing efficiency and reducing human errors in fields like: Healthcare — AI-assisted robotic surgery, robotic nurses for elderly care Construction — 3D-printing robots that build homes faster and cheaper Agriculture — AI-driven machines that optimize farming and reduce waste The Rise of General AI Right now, AI is mostly narrow, meaning it specializes in specific tasks (like ChatGPT processing language). But the race for Artificial General Intelligence (AGI) is on—machines that think, reason, and learn like humans. If AGI becomes reality, it could revolutionize nearly every industry—but it also raises serious ethical concerns. AI & Robotics in Space AI-powered robots will play a huge role in space exploration, from rover missions on Mars to asteroid mining and even building structures on the Moon. NASA, SpaceX, and other space agencies are actively investing in AI to navigate space autonomously and assist human astronauts. 3. The Risks: What Could Go Wrong? While AI and robotics promise a bright future, we must also be aware of the challenges and risks. Job Displacement vs. Job Creation Many manual and repetitive jobs (such as assembly line workers, truck drivers, and customer service agents) will be replaced by AI and automation. However, new jobs in AI oversight, robotics maintenance, and AI ethics will emerge. The challenge? Will people be prepared for this shift? Ethical Dilemmas AI bias and discrimination (e.g., biased hiring algorithms, racial profiling in law enforcement AI). The risk of AI surveillance and loss of privacy. The potential for deepfakes and misinformation spreading unchecked. Safety Concerns Autonomous weapons: AI-powered warfare could make decision-making more dangerous and unpredictable. Over-reliance on AI: What happens when humans become too dependent on AI-driven systems, and they fail? The Call for Regulation To prevent AI from spiraling out of control, governments and organizations must implement AI safety protocols, transparent regulations, and ethical AI development. Without these safeguards, AI could do more harm than good. 4. The Fusion of AI & Robotics: Smarter, More Adaptive Machines One of the most exciting developments is the deepening fusion between AI and robotics. How AI is Making Robots Smarter Learning from experience — Robots can now learn from past mistakes, much like a human. Real-time decision-making — AI-powered robots can adapt to new environments without needing to be reprogrammed. Emotional intelligence — AI-driven humanoid robots are being trained to recognize and respond to human emotions. This AI-robotic fusion will redefine industries, making robots more autonomous, capable, and integrated into our daily lives. 5. A Hopeful Future: The Road Ahead While AI and robotics bring risks, they also offer unparalleled opportunities to improve human life: Healthcare breakthroughs — AI-powered diagnosis, robotic surgeries, and personalized medicine could extend human lifespan. Reducing hazardous jobs — Robots will take on dangerous tasks, keeping humans safe. Enhancing accessibility — AI-driven tech will help people with disabilities, improving mobility and communication. The Key Takeaway AI and robotics will not replace humanity—they will enhance it. Our role? To guide AI development responsibly, ethically, and thoughtfully. As long as we stay ahead of the risks and embrace the opportunities, the future of AI and robotics could be one of the greatest revolutions in human history. “The future is not AI vs. humans. It’s AI with humans.” Final Thoughts: Are We Ready? The AI and robotics revolution is already here. The question is: Are we prepared for what’s coming next? --- ## Beyond the Algorithm: Why Industry Veterans are Still Vital in the Age of AI Consulting - URL: https://webofmike.com/beyond-the-algorithm/ - Markdown source: https://webofmike.com/beyond-the-algorithm/index.md - Author: Mike Moore - Published: 2024-10-29 - Last modified: 2024-10-29 - Tags: AI Consulting, DevOps, Developers, Platform Engineering, Software Consulting - Description: The irreplaceable value of industry veterans in enterprise architecture and technology solutions in the age of AI. - Word Count: 162 In an era where artificial intelligence consulting dominates tech headlines, a thoughtful perspective from ActualHumans reminds us of an often-overlooked truth: the irreplaceable value of industry veterans in enterprise architecture and technology solutions. While AI solutions continue to revolutionize how we work, the deep expertise, intuition, and adaptability of seasoned technology consulting professionals remain crucial for enterprise success. The article makes a compelling case for why human expertise cannot be fully replicated by algorithms. Industry veterans bring decades of domain-specific knowledge enabling navigation of complex DevOps and cloud-native computing challenges with nuanced understanding. Their ability to think creatively, adapt to unprecedented situations, and foster meaningful collaboration distinguishes them from AI-powered systems. These professionals excel where AI typically falls short: ethical decision-making, customizing solutions for unique enterprise needs, and integrating new technologies with legacy systems. This balanced perspective advocates for a synergistic approach combining AI’s data-processing capabilities with industry veterans’ experience and intuition, building more robust, ethical, and effective cloud-native solutions for enterprises. --- ## Unlocking Unlimited Observability: How groundcover's eBPF Powers Full Visibility Without Limits - URL: https://webofmike.com/unlocking-unlimited-observability/ - Markdown source: https://webofmike.com/unlocking-unlimited-observability/index.md - Author: Mike Moore - Published: 2024-08-02 - Last modified: 2024-08-02 - Tags: Observability, Groundcover, eBPF, Logging, Metrics, APM, Monitoring - Description: How groundcover leverages eBPF to provide unlimited logs, metrics, and traces for full observability without data caps. - Word Count: 617 In today’s constantly evolving digital world, ensuring your applications and AI platforms run smoothly is more critical than ever. From user experience to system security, every aspect of your application relies on how well you can monitor its performance. But with the complexity of modern infrastructures, achieving complete observability can be challenging. This is where groundcover, powered by eBPF, comes into play as the ultimate solution for full visibility, offering unlimited logs, metrics, and traces. Why Full Visibility Matters in Observability Observability isn’t just a buzzword—it’s a necessity. As your applications scale and evolve, the ability to monitor everything from performance to user behavior becomes essential. Observability encompasses several key components: Application Performance Monitoring (APM): APM tools help track and manage the performance and availability of software applications. They allow you to detect and diagnose complex application performance issues. Metrics: Metrics provide quantitative data on system performance, such as CPU usage, memory consumption, and response times. High cardinality metrics, which involve a large number of unique combinations of dimensions, offer deep insights into system behavior. Logging: Logs are records of events that happen within your applications and infrastructure. They are invaluable for troubleshooting, security, and auditing. Tracing: Traces follow the flow of requests through your services, helping you pinpoint where slowdowns or errors occur. But as valuable as these tools are, traditional observability platforms often impose limits on the volume of data you can collect and analyze. This restriction forces you to make tough decisions about what to monitor and what to ignore—decisions that could lead to missed issues or incomplete insights. groundcover and eBPF: Revolutionizing Observability Enter groundcover, a game-changer in the observability landscape. groundcover leverages the power of eBPF (Extended Berkeley Packet Filter) to provide a level of visibility that traditional monitoring tools can’t match. eBPF for Comprehensive Monitoring: eBPF is a revolutionary technology that allows you to run sandboxed programs in the Linux kernel without changing the kernel source code or loading kernel modules. This means you can monitor everything happening within your applications and infrastructure with minimal overhead. groundcover’s implementation of eBPF takes this a step further by offering deep insights into system behavior in real-time. Unlimited Logs, Metrics, and Traces: groundcover uses eBPF to unlock unlimited data collection. No more worrying about data caps or sampling—groundcover provides full visibility across your entire stack. OpenTelemetry Integration: groundcover seamlessly integrates with OpenTelemetry, the industry-standard observability framework. This integration allows you to collect and correlate data across different systems and applications, providing a unified view of your infrastructure’s performance. The Benefits of Unlimited Observability with groundcover Proactive Issue Detection: With groundcover, you don’t need to wait for an issue to arise before you start digging into logs or metrics. The unlimited visibility provided by eBPF allows you to detect anomalies and potential problems before they impact your users. Enhanced Security: By capturing all logs and traces, groundcover ensures that no suspicious activity goes unnoticed. Optimized Performance: High cardinality metrics, supported by groundcover, give you the granular data needed to optimize performance. Cost Efficiency: While traditional observability tools may charge based on data volume, groundcover’s eBPF-powered solution allows for unlimited data collection without breaking the bank. groundcover is the Future of Observability In a world where complete observability is the key to maintaining robust, secure, and high-performing applications, groundcover stands out as the best solution. By harnessing the power of eBPF, groundcover provides unmatched visibility, offering unlimited logs, metrics, and traces. Whether you’re focused on improving your APM strategy, optimizing high cardinality metrics, or integrating with OpenTelemetry, groundcover delivers the comprehensive monitoring you need. Don’t let limited visibility hold you back—embrace groundcover for a future where full observability is not just possible, but practical and affordable. --- ## How to Get Ready for the AI Revolution: Embrace, Learn, and Innovate - URL: https://webofmike.com/how-to-get-ready-for-the-ai-revolution/ - Markdown source: https://webofmike.com/how-to-get-ready-for-the-ai-revolution/index.md - Author: Mike Moore - Published: 2024-07-30 - Last modified: 2024-07-30 - Tags: Generative AI, AI Training, AI Tools, AI Education, AI Innovation - Description: Preparing for the AI-driven future through education, experimentation, and strategic partnerships. - Word Count: 510 The advent of generative AI is not something to be feared—it’s an exciting opportunity to elevate your business and career. Getting ready for this AI-driven future involves a blend of embracing new technologies, continuously learning, and fostering an innovative mindset. Here’s how you can prepare effectively and stay ahead in this rapidly evolving landscape. 1. Dive Into AI Education and Training Online Courses and Certifications: Start by enrolling in online courses that cover the basics of AI, machine learning, and generative AI. Platforms like Coursera, edX, and Udacity offer courses designed by leading institutions and industry experts. Workshops and Bootcamps: Many organizations and tech companies offer intensive workshops and bootcamps that provide hands-on experience with AI tools and technologies. Industry Webinars and Conferences: Attend webinars, conferences, and meetups focused on AI and technology. Events like the AI Summit, CES, or industry-specific conferences provide valuable insights into the latest trends and innovations. 2. Experiment with AI Tools and Platforms Explore AI Tools: Familiarize yourself with popular AI tools and platforms that are relevant to your field. For content creation, tools like Jasper or Copy.ai can help generate marketing copy, while platforms like DALL-E and Midjourney offer creative image generation. Implement Small AI Projects: Start small by incorporating AI into manageable projects within your business. Use AI for automating routine tasks, analyzing data, or enhancing customer interactions. Pilot AI Solutions: If you’re considering adopting more advanced AI solutions, conduct pilot projects to test their effectiveness. 3. Foster a Culture of Continuous Learning and Innovation Encourage a Learning Mindset: Promote a culture where continuous learning and adaptability are valued. Create Innovation Labs: Set up innovation labs or dedicated spaces within your organization where employees can experiment with new technologies, including AI. Collaborate with AI Experts: Partner with AI consultants or hire experts who can provide guidance on integrating AI into your business processes. 4. Stay Updated with Industry Trends Subscribe to AI News Sources: Stay informed about the latest developments in AI by subscribing to industry newsletters, blogs, and journals. Join AI Communities: Engage with online AI communities and forums, such as Reddit’s r/MachineLearning or LinkedIn groups focused on AI. Monitor Competitors: Keep an eye on how competitors are leveraging AI. 5. Develop Strategic AI Partnerships Collaborate with AI Startups: Partnering with AI startups can provide access to cutting-edge technology and innovative solutions. Engage with Academic Institutions: Collaborate with universities and research institutions that are conducting AI research. Participate in AI Ecosystems: Join industry associations and ecosystems focused on AI. In Conclusion Preparing for the AI revolution is about more than just adopting new tools; it’s about cultivating a proactive and innovative mindset. By investing in education, experimenting with AI tools, fostering a culture of continuous learning, staying updated with industry trends, and forming strategic partnerships, you can effectively navigate the AI landscape and turn challenges into opportunities. The future is not just about surviving in an AI-driven world; it’s about thriving and leading the way. Embrace the excitement, stay curious, and leverage AI to unlock new possibilities for your business and career. --- ## Revisiting Observability: A Deep Dive Into the State of Monitoring, Costs, and Data Ownership - URL: https://webofmike.com/revisiting-observability/ - Markdown source: https://webofmike.com/revisiting-observability/index.md - Author: Mike Moore - Published: 2024-05-02 - Last modified: 2024-05-02 - Tags: eBPF, Kubernetes, Cloud, SaaS, Observability, DevOps, Platform Engineering, SRE - Description: A deep dive into the current state of observability, covering instrumentation challenges, cost surprises, and data ownership. - Word Count: 847 Originally posted on DZone Hey internet humans! I’ve recently re-entered the world of observability and monitoring after a short detour in the Internal Developer Portal space. Since my return, I have felt a strong urge to discuss the general sad state of observability in the market today. I still have a strong memory of myself, knee-deep in Kubernetes configs, drowning in a sea of technical jargon, not clearly knowing if I’ve actually monitored everything in my stack, deploying heavy agents, and fighting with engineering managers and devs just to get their code instrumented only to find out I don’t have half the stuff I thought I did. Sound familiar? Most of us have been there. The three pain points that are top-of-mind for me these days are: The state of instrumentation for observability The horrible surprise bills vendors are springing on customers and the insanely confusing pricing models that can’t even be calculated Ownership and storage of data - data residency issues, compliance, and control Instrumentation The monitoring community has got a fantastic new tool at its disposal: eBPF. It’s a game-changing tech (a cheat code, if you will) that allows us to trace what’s going on in our systems without all the usual headaches. With eBPF, we can dive deep into the inner workings of applications and infrastructure, capturing data at the kernel level with minimal overhead. I’ve had first-hand experience deploying monitoring solutions at scale during my tenure at companies like Datadog, Splunk, and CA Technologies. I’ve seen the patchwork of APM, infrastructure, logs, OpenTelemetry, custom instrumentation, and open-source solutions that is often patched together (usually poorly) just to try and get at the basics. At this point, there are two things that happen: Not everything is monitored because we have no idea where everything is. We end up with far less than 100% coverage. We start having those cringe-worthy discussions on “should we monitor this thing” due to the sheer cost of monitoring, often costing more than the infrastructure our applications and microservices are running on. OpenTelemetry is fantastic for solving vendor lock-in and has a much larger community working on it, but it takes A LOT OF WORK. It takes real collaboration between all teams to make sure everyone is instrumenting manually and that every single library is well supported. From my observations, this generally results in an incomplete patchwork giving us a very incomplete picture 95% of the time. With proper eBPF deployment and some secret sauce, these core concerns simply don’t have to worry us—as long as there’s a simplified pricing model in place. We can get full-on 360-degree visibility in our environments with tracing, metrics, and logs without the hassle. The Elephant in the Room: Cost and the Awful State of Pricing in the Observability Market Today If I’d have a penny for every time I’ve heard: “I need an observability tool to monitor the cost of my observability tool.” Traditional monitoring tools often come with hefty price tags attached, and often ones that’s a big fat surprise when we add a metric or a log line—especially when it’s at scale! These tools typically charge based on volume of data ingested, and it’s easy to underestimate how quickly those costs add up. I’ve seen customers receive multiple tens of thousands of dollars (sometimes multiple hundreds of thousands) in overage bills because some developer added a few extra log lines or because someone needed additional cardinality in a metric. Those costs are very real for very simple mistakes, especially when often there are no controls in place to keep them from happening. That’s when a modern solution should step in to save the day. By offering transparent pricing based on usage—not volume, ingest, egress, or some unknown metric you have no idea how to calculate—we should be able to get specific about the cost of monitoring and set clear expectations knowing we can see everything end-to-end without sacrificing because the cost may be too high. Ownership and Storage of Data The next topic I’d like to touch upon is the importance of data residency, compliance, and security in the realm of observability solutions. In today’s business landscape, maintaining control over where and how data is stored and accessed is crucial. Various regulations, such as GDPR, require organizations to adhere to strict guidelines regarding data storage and privacy. Traditional cloud-based observability solutions may present challenges in meeting these compliance requirements, as they often store data on third-party servers dispersed across different regions. Opting for an observability solution that allows for on-premises data storage addresses these concerns effectively. By keeping monitoring data within the organization’s data center, businesses gain greater control over its security and compliance. This approach minimizes the risk of unauthorized access or data breaches, thereby enhancing data security and simplifying compliance efforts. For organizations seeking to ensure compliance, enhance data security, and optimize costs, an observability solution that facilitates on-premises data storage offers a compelling solution. By maintaining control over data residency and security while achieving cost efficiencies, businesses can focus on their core competencies and revenue-generating activities with confidence. --- ## Mastering Kubernetes: Key Metrics for Cluster Monitoring - URL: https://webofmike.com/mastering-kubernetes-key-metrics/ - Markdown source: https://webofmike.com/mastering-kubernetes-key-metrics/index.md - Author: Mike Moore - Published: 2024-04-14 - Last modified: 2024-04-14 - Tags: Kubernetes, Observability, Monitoring, DevOps, Platform Engineering, SRE - Description: A deep dive into the essential metrics for effective Kubernetes cluster monitoring. - Word Count: 75 Kubernetes administration is like navigating through a perpetually expanding puzzle, where distinguishing essential components from duplicate information presents ongoing challenges. Beyond surface-level monitoring lies the opportunity to extract meaningful signals from container telemetry, pod performance, CPU and memory utilization, API interactions, and environmental metrics. Mastery comes from transforming raw data into strategic insights that optimize performance, improve efficiency, and strengthen infrastructure resilience. Embrace the complexity as a pathway toward operational proficiency in container orchestration environments. --- ## Maximizing DevOps Efficiency: Best Practices, KPIs, and Realtime Feedback - URL: https://webofmike.com/maximizing-devops-efficiency/ - Markdown source: https://webofmike.com/maximizing-devops-efficiency/index.md - Author: Mike Moore - Published: 2023-03-08 - Last modified: 2023-03-08 - Tags: DevOps, Developer Portal, Developer Experience, Platform Engineering - Description: How to adopt best practices for DevOps teams using automation, version control, and developer portals for real-time feedback. - Word Count: 558 Inefficient DevOps practices can lead to reduced productivity, delayed releases, and increased costs. A Developer Portal can help teams understand best practices, KPIs, and provide real-time feedback to optimize performance. As a DevOps team, it’s important to adopt best practices to ensure that your organization can deliver high-quality software products efficiently and reliably. Let’s take a closer look at some of the most important best practices for DevOps teams, as well as the risks of not implementing them. One of the key best practices for DevOps teams is to use automation wherever possible. By automating repetitive tasks like testing and deployment, you can reduce the risk of errors and save time that can be spent on more valuable work. Additionally, automation can help improve consistency and standardization across your organization, reducing the risk of inconsistencies or errors caused by manual processes. Another important best practice is to use version control for your codebase. Version control allows you to keep track of changes to your code over time, which can be invaluable for debugging and collaboration. It also enables you to roll back changes if necessary and can help ensure that everyone on your team is working with the most up-to-date code. Implementing these best practices can help your DevOps team work more effectively and efficiently, but what happens if you don’t adopt them? The risks can be significant. Without automation, for example, you may find that your team spends more time than necessary on manual tasks, reducing productivity and increasing the risk of errors. And without version control, it can be difficult to track changes to your code, leading to confusion and potential conflicts between team members. In addition, failing to adopt best practices can make it harder to scale your operations as your organization grows. As you take on more complex projects and work with larger teams, the need for automation and version control becomes even more critical. An Internal Developer Portal provides a central hub for developers to access resources, documentation and support related to DevOps best practices. The portal can help educate developers on best practices for things like continuous integration and deployment, automated testing, and monitoring. By providing access to this information, developers can stay up-to-date on the latest industry trends and improve their skills. Additionally, the portal can help developers identify key performance indicators (KPIs) that are important to their team’s success, such as deployment frequency, mean time to recovery (MTTR), and code quality metrics. The Developer Portal can also provide real-time feedback to developers and engineering leadership. For example, it can be integrated with tools like Jenkins, GitHub, and JIRA to provide visibility into development workflows and performance metrics. This visibility can help identify areas where improvements can be made, such as reducing the number of failed builds or improving the speed of deployments. A Developer Portal can play a critical role in promoting DevOps best practices, improving team performance, and increasing efficiency. By providing access to resources, education, and real-time feedback, developers can continually improve their skills and processes, and engineering leaders can make informed decisions to optimize team performance. Adopting best practices is essential for DevOps teams that want to deliver high-quality software products efficiently and reliably. By using automation and version control, you can reduce the risk of errors, save time, and improve consistency and standardization across your organization. --- ## Open Source Solutions: Cost-Saving or Free Like a Puppy? - URL: https://webofmike.com/open-source-cost-saving-or-free-like-a-puppy/ - Markdown source: https://webofmike.com/open-source-cost-saving-or-free-like-a-puppy/index.md - Author: Mike Moore - Published: 2023-03-06 - Last modified: 2023-03-06 - Tags: Open Source Software, Free Software, Community Development, Maintenance Costs, Security Risks - Description: Open-source software may be free, but it can come with significant maintenance and security risks that businesses should consider. - Word Count: 521 Open-source software may be free, but it can come with significant maintenance and security risks that businesses should consider. While highly effective in some situations, it may not be the best option for specialized software or high levels of support. So, you’ve heard of open-source software, right? It’s that free software that’s developed by a bunch of tech enthusiasts in their free time. But, have you ever stopped to think about why it’s free? Well, let me tell you friends and already offended internet folks, “open-source software is free in the same way that a puppy is free.” When you adopt a puppy, you quickly realize that it requires a lot of time, attention, and resources to properly take care of it. Similarly, when you adopt open-source software, you’re essentially adopting a community of volunteers who develop and maintain the software. These volunteers are passionate about technology, but they’re not necessarily focused on meeting the needs of businesses. This means that if you want to use open-source software in a business setting, you may need to invest significant time and resources into maintaining and supporting it yourself. Now, don’t get me wrong, open-source software can be highly effective in certain situations. But, it may not be the best option for businesses that require specialized software or high levels of support and maintenance. Commercial software vendors often provide dedicated support teams, regular updates and patches, and a high level of customization and integration with other software. Another thing to consider is security. While open-source software is often highly secure and reliable, it can also be vulnerable to security risks if it’s not properly maintained and updated. In some cases, commercial software may be more secure and reliable than open-source software because it’s developed and maintained by a dedicated team of security experts. In addition to the potential maintenance and security risks of open-source software, there’s also a risk that the project may try to monetize its users and upsell them on poorly supported software. This can be particularly challenging for businesses that rely heavily on the software, as they may become locked into an inferior solution. I’ve watched countless enterprises spin their wheels, claim they’re evaluating for “several months” having made little-to-no progress, or having had to customize this so much this now becomes a non-revenue generating product their building, maintaining, and supporting all in the name of “because it’s open source and doesn’t cost us anything”. Now, I don’t know who their CxOs are and why they don’t look at them square in the face and say: “Nope” but I suspect it’s because it gets hidden down among the lower layers, and the real true cost of this never gets spoken about to the people who really need to understand it. So, the bottom line is that open-source software is free like a puppy. You can take it home for free, but it’s going to require a lot of time and resources to properly take care of it. Before adopting open-source software for your business, it’s important to carefully consider your options and evaluate the costs and benefits of each option. --- ## Maximizing Your Developer Efficiency: Scaffolding for Faster Time to Market - URL: https://webofmike.com/maximizing-developer-efficiency-scaffolding/ - Markdown source: https://webofmike.com/maximizing-developer-efficiency-scaffolding/index.md - Author: Mike Moore - Published: 2023-02-10 - Last modified: 2023-02-10 - Tags: Developer Portal, Developer Experience, Platform Engineering - Description: How scaffolding tools help developers work more efficiently and deliver software faster. - Word Count: 390 As software development continues to evolve, developers are faced with the challenge of delivering high-quality applications at an ever-increasing pace. The pressure to deliver more and faster can be overwhelming, especially for teams working on large, complex projects. However, with the rise of low-code platforms and scaffolder tools, developers can now achieve faster time to productivity and focus on delivering high-quality software. What is a Scaffolding Tool? A scaffolder tool is a software application that generates the boilerplate code for a project, saving developers time and effort in writing code from scratch. The generated code serves as a starting point, providing a foundation for developers to build upon. With a scaffolder tool, developers can focus on writing the custom code required to bring their vision to life, rather than spending countless hours writing the same repetitive code that every project requires. The Benefits of Using a Scaffolder Tool Increased Efficiency and Speed By generating the boilerplate code, kicking off infrastructure builds, attaching tooling, etc for a project, scaffolder tools help developers work more efficiently and get up to speed faster. This increased speed means developers can deliver software faster, meeting tight deadlines and reducing time to market. Improved Developer Experience Scaffolder tools can also enhance the developer experience by providing a simple, streamlined interface for creating and managing projects. This can help developers stay organized, focused, and on track, reducing the risk of mistakes and rework. Enhanced Internal Developer Portal An internal developer portal is an essential tool for managing and sharing information among developers within an organization. A scaffolder tool can help enhance the internal developer portal by providing a centralized repository for boilerplate code and templates, making it easier for developers to find what they need and get up to speed quickly. Low-Code Approach to Development The low-code approach to development has been gaining traction in recent years, and for good reason. By leveraging a scaffolder tool, developers can write less code, freeing up time and resources to focus on the custom code that sets their software apart. Overall, great scaffolder tools offer a range of benefits to developers, from faster time to productivity to improved developer experience. With the rise of low-code platforms, scaffolder tools are becoming increasingly popular and provide a simple, streamlined way to get up to speed quickly and focus on delivering high-quality software. --- ## Boosting Your Developer Onboarding Efficiency: Why Investing in a Developer Portal is Smarter Than Building Your Own - URL: https://webofmike.com/boosting-developer-onboarding-efficiency/ - Markdown source: https://webofmike.com/boosting-developer-onboarding-efficiency/index.md - Author: Mike Moore - Published: 2023-02-08 - Last modified: 2023-02-08 - Tags: Developer Portal, Developer Experience, API - Description: Why investing in a developer portal solution is smarter than building your own for API documentation and developer onboarding. - Word Count: 519 APIs (Application Programming Interfaces) play a crucial role in modern software development, allowing different systems and applications to communicate and exchange data with each other. To ensure that APIs are used effectively and efficiently, it’s important to provide comprehensive documentation that developers can reference. Why GOOD Documentation is Important Good API documentation acts as a reference guide for developers, helping them understand how to use an API, what it does, and what to expect from it. Comprehensive documentation helps to reduce friction in the integration process, speeds up the development process, and ensures that APIs are used correctly. Types of API Documentation There are several types of API documentation, including reference documentation, tutorials, code samples, and sample applications. Reference documentation provides a technical description of an API, including information about the API’s methods, parameters, and return values. Tutorials provide step-by-step guidance for using an API, and code samples and sample applications demonstrate how an API can be used in a real-world context. The Role of Internal Developer Portals An internal developer portal is a central location for API documentation and other resources that are relevant to developers. These portals can be hosted internally, or they can be built using a cloud-based API portal solution. Benefits of Internal Developer Portals Internal developer portals provide a number of benefits, including increased visibility, easier collaboration, and improved API discoverability. By centralizing API documentation and resources in one location, developers can easily find what they need and collaborate more effectively with their peers. Key Features of Internal Developer Portals There are several key features that should be included in an internal developer portal, including a searchable API catalog, detailed reference documentation, code samples and tutorials, and a forum or Q&A section where developers can ask questions and get answers. How to Create an Internal Developer Portal Creating an internal developer portal can be a complex process, but it can also be a valuable investment in the long-term success of your API program. Building and maintaining your own developer portal can be time-consuming and resource-intensive. On the other hand, buying a developer portal solution provides you with a ready-to-use platform that has been optimized for performance and user experience. You can focus on delivering value to your developers without worrying about the nitty-gritty of platform management. Best Practices for Documenting APIs When documenting your APIs, it’s important to follow best practices to ensure that your documentation is comprehensive, accurate, and easy to understand. Some best practices include using clear and concise language, providing code samples and tutorials, and including detailed reference documentation for each API. The Role of Developer Engagement In order to ensure that your internal developer portal is successful, it’s important to engage with your developer community. This can be done by hosting regular meetups, workshops, and other events, as well as providing ongoing support and resources through your developer portal. Internal developer portals play a crucial role in ensuring that APIs are used effectively and efficiently. By providing comprehensive documentation and resources, developer portals help to reduce friction in the integration process, speed up the development process, and increase API adoption. --- ## Maximizing Developer Onboarding for Improved Retention and Growth: A Comprehensive Guide - URL: https://webofmike.com/maximizing-developer-onboarding/ - Markdown source: https://webofmike.com/maximizing-developer-onboarding/index.md - Author: Mike Moore - Published: 2023-02-06 - Last modified: 2023-02-06 - Tags: Platform Engineering, DevOps, Developer Experience - Description: A comprehensive guide to structuring developer onboarding for improved retention and productivity. - Word Count: 223 The onboarding experience sets the tone for new developers and shapes their initial perception of the organization and role. A well-structured program should equip developers with necessary tools and information for autonomy and success. Setting Goals and Working Backwards Effective onboarding begins by reflecting on your own early experiences as an employee. Determine what developers should accomplish by the end of their onboarding period—whether handling tickets, contributing to architecture, or pushing code. Consider codebase complexity and realistic timelines. Breaking the process into phases with daily objectives helps developers track progress visually. Building Team Connections During the first few days, prioritize making new developers feel welcomed, particularly remote hires. Schedule an introductory call with team members and managers on day one. Arrange individual one-on-one meetings throughout the first week with experienced team members. Managers should conduct regular check-ins (two to three times weekly initially, scaling back afterward) to assess adaptation and work style. These meetings significantly boost productivity in smaller teams. Knowledge Transfer Sessions These sessions are fundamental, providing comprehensive understanding of the codebase and product. Meetings with cross-functional teams and project managers clarify roles, client expectations, and product features. Fellow developers can contextualize the codebase, facilitating connections and insights during coding work. Using Developer Platforms Developer portals streamline onboarding by reducing time-to-value and increasing ROI through improved efficiency and access to training resources. --- ## Mastering the Art of Platform Engineering: The Latest Secret to Digital Transformation - URL: https://webofmike.com/platform-engineering/ - Markdown source: https://webofmike.com/platform-engineering/index.md - Author: Mike Moore - Published: 2023-02-02 - Last modified: 2023-02-02 - Tags: Platform Engineering, DevOps, Developer Experience, DevX, Developer Portal - Description: How platform engineering is the future of technology operations and a key driver of organizational success. - Word Count: 391 As technology continues to evolve and disrupt traditional business models, organizations are constantly seeking ways to streamline and optimize their operations. This is where platform engineering comes in. But what exactly is platform engineering? In simple terms, it’s a discipline that focuses on the creation, deployment, and management of platforms and tools that enable the development and delivery of software products. It’s no secret that DevOps has been a hot topic in recent years, and many companies have embraced this methodology to automate and streamline their software development and delivery processes. Platform engineering can be seen as an extension of DevOps, but with a broader focus on creating a more holistic platform for technology operations. So, how does platform engineering differ from DevOps? While both disciplines share a common goal of improving software delivery, DevOps focuses primarily on the development and operations of individual software products, while platform engineering takes a more systemic approach by creating a shared platform for multiple software products and teams. The demand for platform engineering solutions has skyrocketed in recent years due to the increasing need for organizations to move quickly and efficiently in the digital age. A well-designed platform can provide a competitive advantage by enabling faster and more reliable software delivery, reducing operational costs, and improving overall organizational efficiency. From a technical perspective, platform engineering can provide numerous benefits such as reduced operational overhead, improved resource utilization, and better collaboration across teams. For executive business decision makers, platform engineering can lead to improved time-to-market, increased efficiency, and reduced costs. Visibility is a crucial aspect of platform engineering, as it allows teams to have a comprehensive understanding of the platform and its components. This can improve problem resolution times, reduce downtime, and provide valuable insights into the platform’s performance and usage. Investing in a great platform engineering solution can provide tremendous value to your organization by improving operational efficiency, reducing downtime, and increasing productivity. With a well-designed platform, your teams will be able to move quickly and efficiently, allowing you to stay ahead of the competition in the digital age. In conclusion, platform engineering is the future of technology operations and a key driver of organizational success in the digital age. By embracing this discipline, organizations can streamline their operations, improve software delivery, and stay ahead of the curve in the constantly evolving tech landscape. --- ## Maximize your ROI with a Developer Portal: How Improving DevOps, DevX, and SRE Can Drive Visibility, Productivity, and Profitability - URL: https://webofmike.com/developer-portal/ - Markdown source: https://webofmike.com/developer-portal/index.md - Author: Mike Moore - Published: 2023-01-31 - Last modified: 2023-01-31 - Tags: Developer Portal, IDP, Developer Experience, DevX, Platform Engineering, DORA, SRE - Description: How a well-implemented developer portal can impact DevOps processes, developer experience, and overall efficiency. - Word Count: 141 Implementing and maintaining a developer portal involves complexity, but a fully-featured and well-implemented developer portal can have a significant impact on your DevOps processes, developer experience (DevX), site reliability engineering (SRE), and overall efficiency. Technical Benefits Streamlined workflows reduce onboarding time Enhanced collaboration across teams Increased visibility through centralized documentation Improved support via FAQs and resources Business Benefits Higher productivity and reduced costs Better customer experience Increased product adoption Improved user engagement Strategic Value A well-designed portal gives leadership a bird’s eye view of the entire development process, enabling informed decision-making. It centralizes resources, accelerating project timelines and improving profitability. Build vs. Buy Analysis Custom solutions typically cost more long-term than pre-built platforms. Organizations often waste resources on solutions that don’t generate revenue and distract from core competencies. Pre-built platforms offer lower total cost of ownership, faster implementation, and superior usability. --- ## Thwarting Innovation with Entrenched Mindsets - URL: https://webofmike.com/innovation-failure/ - Markdown source: https://webofmike.com/innovation-failure/index.md - Author: Mike Moore - Published: 2023-01-30 - Last modified: 2023-01-30 - Tags: Thoughts, Leadership, Management, Innovation, Mindset - Description: How entrenched mindsets and resistance to change stifle innovation in organizations. - Word Count: 457 In today’s fast-paced business environment, innovation is key to staying ahead of the competition. However, many organizations struggle with promoting and encouraging innovation due to the prevalence of entrenched mindsets that stifle creativity and progress. The phrase “that’s just the way it is” and its variants are often heard in various industries, indicating a lack of motivation to change and a tendency to accept the status quo. Entrenched Mindsets as Innovation Killers The mindset of “that’s just the way things are” demonstrates a failure to empower individuals to drive change and innovate. It reflects a lack of motivation and a path of least resistance, resulting in stagnation. Innovation and change are closely tied, and if we remain content with the status quo, there can be no progress. When employees don’t feel empowered to act and make changes, they are likely to become complacent and accept things as they are, without striving for improvement. This can be particularly problematic in organizations that define themselves as innovative or forward-thinking. “Change is too hard” is another problematic mindset, indicating a reluctance to challenge the status quo. This phrase is often used as an excuse for not attempting to make improvements or try new things. However, change is often the key to progress and success. Embracing Change for Innovation Challenging the status quo, taking risks, and embracing the unknown are essential for innovation. Stepping outside of one’s comfort zone and embracing the possibility of failure can lead to learning, adaptation, and improvement. Innovation requires the freedom to explore new ideas without the constraints of assumptions and established processes. This requires a willingness to take risks, embrace change, and look for new solutions, even if it means failing and trying again. Leading by example is crucial in promoting a culture of innovation. When senior leaders demonstrate a willingness to embrace change and take risks, it sets a positive tone for the rest of the organization. The Revenue and Productivity Impact An entrenched mindset can have a significant negative impact on both revenue and productivity within an organization. A lack of motivation to change and complacency with the status quo stifles innovation, which is a key driver of growth and competitiveness. When employees and teams are not empowered to challenge assumptions and explore new ideas, they are unable to improve and adapt to changing market conditions. This can lead to missed opportunities for revenue generation and increased efficiency, as well as a decline in overall productivity levels. Conclusion Overcoming the mindset of “that’s just the way it is” and “change is too hard” is essential for promoting innovation and continuous improvement in organizations. By embracing change, taking risks, and leading by example, leaders can create a culture that values creativity, experimentation, and progress. --- ## DORA Metrics: Understanding the Importance for Developers, SRE, DevOps, and Platform Engineering - URL: https://webofmike.com/dora-metrics/ - Markdown source: https://webofmike.com/dora-metrics/index.md - Author: Mike Moore - Published: 2023-01-30 - Last modified: 2023-01-30 - Tags: DORA, Developers, DevOps, Developer Experience, Developer Portal - Description: Understanding DORA metrics and their importance for improving DevOps processes and software delivery. - Word Count: 332 Have you ever wondered how successful organizations manage to deliver high-quality software at lightning speeds? The secret lies in DevOps practices and key performance indicators known as DORA metrics. DORA stands for DevOps Research and Assessment. These metrics were developed by Dr. Nicole Forsgren, Jez Humble, and Gene Kim through the State of DevOps report, compiled from data provided by over 30,000 technical professionals worldwide. The Four Key DORA Metrics Lead Time: Time from code commitment to successful production deployment Deployment Frequency: How often code deploys to production (daily, weekly, or monthly) Mean Time to Recovery (MTTR): Average recovery time from production incidents Change Failure Rate: Percentage of production changes causing failures or outages Why DORA Metrics Matter These metrics offer valuable insights into the speed and quality of software delivery. By measuring DevOps effectiveness, organizations identify improvement areas and benchmark performance against industry standards. Each metric serves distinct purposes: Lead time reveals delivery bottlenecks Deployment frequency increases release speed and reliability MTTR reduces system downtime Change failure rate improves code quality and reduces outage risk Developer Experience Impact DORA metrics significantly influence developer experience. The performance and efficiency of the software delivery process has a direct impact on the productivity and satisfaction of developers. Optimizing these metrics ensures developers access necessary tools and resources, fostering engagement, collaboration, and innovation while supporting talent retention. Real-World Success Stories A technology company reduced lead time by 50% and increased deployment frequency by 200% using DORA metrics. A financial services organization decreased MTTR by 70% and nearly eliminated change failures, improving customer satisfaction and system health. The Role of Developer Portals Effective DORA metric tracking requires a Developer Portal—a centralized platform providing tools, resources, and information for developer productivity. This enables real-time metric visualization and contextual analysis with other performance indicators like code quality and security. Conclusion DORA metrics are essential for improving DevOps processes. Whether organizations are tech giants or startups, implementing these metrics demonstrates tangible benefits for staying competitive in today’s digital landscape. --- ## Getting Started with FDM 3D Printing - URL: https://webofmike.com/fdm-3d-printing/ - Markdown source: https://webofmike.com/fdm-3d-printing/index.md - Author: Mike Moore - Published: 2021-07-22 - Last modified: 2021-07-22 - Tags: 3D Printing - Description: A beginner's guide to FDM 3D printing, covering printer selection, filaments, and materials. - Word Count: 169 Contemplating a 3D printer purchase? This guide covers the fundamentals of FDM technology and practical considerations for beginners. What is FDM? Fused deposition modeling (FDM), also known as fused filament fabrication (FFF), is simply the most widely used type of 3D printing at a consumer level. The process involves pushing thermoplastic filaments—including PLA, PETG, and ABS—through a heated nozzle. Material melts and applies in successive layers to a build platform until the object is complete. Selecting Your Printer I recommend Prusa Research printers—they offer quality construction kits enabling users to learn printer mechanics through assembly. Creality provides a more budget-friendly alternative. Filament Considerations Initial setup requires exploring various materials and colors. Optional upgrades like the Prusa MMU2S enable multi-color printing capabilities. Choosing the Right Material PLA: Suitable for toys, decorative items, and quick prototypes. Benefits include low printing temperatures, vibrant colors, and biodegradability. PETG: Better suited for durable objects requiring higher temperature resistance and outdoor use. Additional materials like ABS and water-soluble support filaments warrant exploration as skills develop. --- ## WordPress Optimization: Installing and Configuring WordPress - URL: https://webofmike.com/installing-and-configuring-wordpress/ - Markdown source: https://webofmike.com/installing-and-configuring-wordpress/index.md - Author: Mike Moore - Published: 2019-01-04 - Last modified: 2019-01-04 - Tags: Tutorials, Code - Description: Part 5 of the WordPress optimization series - installing MySQL and WordPress on a Digital Ocean droplet. - Word Count: 193 Welcome to part 5 of 8 in a series on Kick-Ass WordPress Optimization. This guide covers database and WordPress installation on a Digital Ocean droplet following server and DNS setup. Installing MySQL and Configuring Your Database Actually Installing MySQL The foundation for WordPress requires a database to store posts, settings, and content. Installation begins with a console command: sudo apt-get -y install mysql-server During installation, you’ll be prompted to set a root password—this is critical to record for future access. Creating a MySQL Database After logging into MySQL with your root credentials, create a database with a command like: CREATE DATABASE yourblog; Creating a MySQL User Rather than using the root account (a security risk), establish a dedicated user: CREATE USER 'yourblog'@'localhost' IDENTIFIED BY 'yourpassword'; Granting User Permissions Grant your new user database access: GRANT ALL PRIVILEGES ON yourblog.* TO 'yourblog'@'localhost' WITH GRANT OPTION; Tuning MySQL for WordPress Optimize performance by editing /etc/mysql/my.cnf and adding InnoDB configuration parameters for buffer pooling, flush methods, and file-per-table settings. Restart MySQL to apply changes. Installing and Configuring WordPress Use the wget utility to download WordPress files directly to your server, then follow the standard installation wizard. --- ## WordPress Optimization: Configuring CloudFlare as Your DNS Provider - URL: https://webofmike.com/wordpress-optimization-configuring-cloudflare/ - Markdown source: https://webofmike.com/wordpress-optimization-configuring-cloudflare/index.md - Author: Mike Moore - Published: 2019-01-04 - Last modified: 2019-01-04 - Tags: Tutorials, Code - Description: Part 4 of the WordPress optimization series - setting up CloudFlare as your DNS provider. - Word Count: 200 Welcome to Part 4 of an 8-part series on implementing CloudFlare as a DNS provider for WordPress optimization. This guide assumes readers have either set up their own web server or use external hosting. What is CloudFlare? CloudFlare functions as a content delivery network and security service. It protects and accelerates any website online by routing traffic through a global network that optimizes page delivery while blocking threats and malicious bots. The service has demonstrated significant performance improvements for websites using its platform. Step 1: Creating a CloudFlare Account Sign up at cloudflare.com with just an email address and username. You’ll need your domain name to proceed. Step 2: Importing Your DNS Records Enter your domain name during setup, and CloudFlare automatically retrieves your existing DNS records. Configure your DNS settings with your domain and corresponding IP address, then save the changes. CloudFlare will provide two nameservers that you’ll need for the next phase. Step 3: Updating Domain Configuration Access your domain registrar (such as GoDaddy) and navigate to domain management settings. Locate your domain and select the option to manage nameservers. Choose “Custom” nameserver configuration and input the two nameservers CloudFlare provided. Save your changes to complete the setup. --- ## Why You Should Use UUIDs in Your APIs - URL: https://webofmike.com/why-you-should-use-uuids-in-your-apis/ - Markdown source: https://webofmike.com/why-you-should-use-uuids-in-your-apis/index.md - Author: Mike Moore - Published: 2019-01-04 - Last modified: 2019-01-04 - Tags: API, Thoughts - Description: Why using UUIDs instead of sequential IDs strengthens API security through obfuscation. - Word Count: 161 What is a UUID? A UUID is a 128-bit identifier represented as 32 hexadecimal digits in the format 8-4-4-4-12, totaling 36 characters. An example is 123e4567-e89b-12d3-a456-426655440000. The total number of possible UUIDs is approximately 3.4 x 10^38, providing virtually unlimited unique identifiers. Why Use UUIDs? The primary advantage centers on security. By using difficult-to-guess identifiers instead of sequential numbers (1, 2, 3, etc.), developers establish a fundamental security layer for their APIs. Consider the contrast: Sequential: http://my.api/resource/12345 UUID-based: http://my.api/resource/123e4567-e89b-12d3-a456-426655440000 The UUID approach makes enumeration attacks substantially harder. Developers can maintain sequential IDs internally for database indexing without exposing them publicly. Addressing Downsides Readability: While humans find UUIDs less readable, this is not a valid reason to abandon them given security benefits. Database Performance: Caching strategies and correlating UUIDs with internal sequential IDs mitigate performance concerns while maintaining security advantages. Conclusion Implementing UUIDs as resource identifiers significantly strengthens API security through obfuscation while remaining practically viable through caching and intelligent database design. --- ## Using A Cache to Reduce Your API Response Time - URL: https://webofmike.com/using-a-cache-to-reduce-your-api-response-time/ - Markdown source: https://webofmike.com/using-a-cache-to-reduce-your-api-response-time/index.md - Author: Mike Moore - Published: 2019-01-04 - Last modified: 2019-01-04 - Tags: API, Microservices - Description: Strategies for implementing caching to improve API performance and developer experience. - Word Count: 129 This presentation covers strategies for implementing caching to improve API performance. Using a cache as a method to reduce your response time requires understanding both when and why caching should be employed. A critical consideration is avoiding misuse of caching technology, which could paradoxically harm performance or cause data loss. When implemented thoughtfully, however, caching delivers substantial speed improvements and reduces service load. Key aspects covered include: When to Cache: Evaluate resource usage frequency, database performance strain, and redundant requests for identical or similar information Technology Options: Various caching solutions and their appropriate use cases Performance Factors: Proper implementation requires balancing cache benefits against potential complications Strategic caching decisions can make your APIs faster, better, and truly make your developers happier by enhancing the developer experience through improved responsiveness. --- ## Full-Stack Kubernetes Monitoring - URL: https://webofmike.com/kubernetes-monitoring/ - Markdown source: https://webofmike.com/kubernetes-monitoring/index.md - Author: Mike Moore - Published: 2019-01-04 - Last modified: 2019-01-04 - Tags: Kubernetes, Docker, Tutorials, Datadog, Monitoring - Description: A practical guide to achieving full-stack visibility in Kubernetes clusters using Datadog. - Word Count: 109 Getting comprehensive visibility into a Kubernetes cluster requires gathering data across multiple layers. This involves collecting information about nodes, pods, applications, and cluster state while considering three key observability pillars: Infrastructure, Applications, and Logs. The core value proposition centers on correlation between these pillars. By enabling teams to connect insights across layers, organizations can improve their Mean-time-to-detection (MTTD) and Mean-time-to-resolution (MTTR)—ultimately enhancing customer satisfaction and user experience. This presentation showcases practical approaches to instrument Kubernetes infrastructure and applications, providing deep and wide visibility into operational and development aspects of our applications. These findings were shared at a Datadog community event in Atlanta, featuring local practitioners discussing monitoring best practices. --- ## Enabling Real-time Analytics with FPGAs - URL: https://webofmike.com/fpga-analytics/ - Markdown source: https://webofmike.com/fpga-analytics/index.md - Author: Mike Moore - Published: 2019-01-04 - Last modified: 2019-01-04 - Tags: Tutorials, Cloud, Thoughts, Leadership, Code - Description: How FPGAs can revolutionize real-time analytics by blurring the lines between hardware and software. - Word Count: 345 With the surge in analytics and big data, organizations need efficient methods to process massive data volumes from IoT devices, mapping systems, and consumer interaction insights. Traditional platforms like Hadoop and Cloudera have provided adequate solutions, yet processing data in-motion with real-time capabilities remains challenging. Blurring the Lines Between Hardware and Software Field Programmable Gate Arrays (FPGAs) represent reconfigurable silicon gate blocks configured through high-level languages like Verilog or VHDL. These devices compile into portable bit-files enabling hardware-level performance for specific functions. FPGAs power H.264 television decoding, high-speed trading algorithms, Tesla vehicle control systems, and F-35 monitoring capabilities. By translating software into gate-level silicon operations, developers achieve near-ASIC performance levels. This fusion creates powerful analytics capabilities where distinguishing software from hardware becomes increasingly difficult. Optimizing for Computation and Real-Time Processing Traditional CPUs process assembly instructions sequentially through multiple steps, creating computational overhead. A 100K instruction program requiring five processing steps per instruction demands approximately 500K clock cycles. While pipelining improves efficiency, CPUs remain optimized for general-purpose computing. FPGAs eliminate this overhead through specialized code blocks performing only necessary functions. Their reconfigurable architecture enables true parallelism—unlike CPU multitasking—delivering orders of magnitude performance improvements. By offloading computationally intensive analytics functions to FPGAs, organizations can process data near real-time or while data remains in-motion. Challenges to Hardware-Accelerated Analytics Cloud migration presents obstacles for FPGA deployment, as these devices require physical hosting infrastructure. Additionally, FPGAs demand skilled engineers and data scientists capable of identifying which algorithms warrant acceleration and optimizing implementation. Open Platforms for Hardware-Accelerated Analytics APIs provide the solution for democratizing FPGA access. By wrapping hardware-accelerated platforms with standardized interfaces, organizations enable broader adoption while abstracting technical complexity. This approach allows diverse systems and third-party consumers to interact with powerful technology safely and predictably. Final Thoughts Despite cloud enthusiasm, underlying technologies deserve focus. Future FPGA solutions might leverage common appliances for frequently used functions or reconfigure platforms at design-time for specific needs. Innovation addressing computational challenges thoughtfully could revolutionize analytics across cancer research, artificial intelligence, life sciences, and data-intensive domains where current processing remains prohibitively slow or expensive. --- ## Datadog Squid Proxy Example for Agent Proxies - URL: https://webofmike.com/datadog-squid-proxy/ - Markdown source: https://webofmike.com/datadog-squid-proxy/index.md - Author: Mike Moore - Published: 2019-01-04 - Last modified: 2019-01-04 - Tags: Datadog, Code, Monitoring - Description: Using SQUID proxy to route Datadog Agent data from restricted network environments. - Word Count: 64 SQUID proxies serve as a practical solution to get your data out of your datacenter when you have a number of security restrictions. This approach enables Datadog Agent communication from hosts within isolated networks that lack direct internet connectivity. Consult the official Datadog documentation for detailed setup instructions—specifically the Agent Proxy Configuration documentation for comprehensive guidance on configuring agents to communicate through proxy servers. --- ## Custom Shell Check for Datadog Agents - URL: https://webofmike.com/custom-shell-check-for-datadog-agents/ - Markdown source: https://webofmike.com/custom-shell-check-for-datadog-agents/index.md - Author: Mike Moore - Published: 2019-01-04 - Last modified: 2019-01-04 - Tags: Datadog, Monitoring, Tutorials, Code - Description: A custom Datadog check that executes shell commands and publishes results as metrics. - Word Count: 92 The Datadog agent stands out for its extensibility. While many out-of-the-box integrations exist, edge cases often require custom solutions. This article presents a standalone custom check that executes shell commands and retrieves numeric values from their output, publishing results to Datadog as metrics named shell.your_metric_name at regular agent check intervals. The check overcomes versioning challenges where previous implementations relied on agent-included code that changed across updates. This rewritten version operates independently. The complete code is available at the GitHub repository: themsquared/datadog-checks/shell. For configuration and implementation guidance, consult the official Datadog Agent documentation. --- ## Custom File Check for Datadog Agents - URL: https://webofmike.com/custom-file-check-for-datadog-agents/ - Markdown source: https://webofmike.com/custom-file-check-for-datadog-agents/index.md - Author: Mike Moore - Published: 2019-01-04 - Last modified: 2019-01-04 - Tags: Datadog, Monitoring - Description: A custom Datadog check that monitors file existence and captures file age metrics. - Word Count: 93 One of the most valuable aspects of the Datadog agent is its extensibility. Although the platform provides numerous out-of-the-box integrations, edge cases frequently emerge that demand custom solutions. This article presents a straightforward custom check that monitors file existence. When a target file is found, the check reports success; if absent, it logs an error tagged with the filename. Additionally, the solution captures file.age and file.modified metrics when files exist, both tagged by filename for easy identification and aging tracking. The implementation is available in the author’s GitHub repository at the file-check directory. --- ## Creation... - URL: https://webofmike.com/creation/ - Markdown source: https://webofmike.com/creation/index.md - Author: Mike Moore - Published: 2019-01-04 - Last modified: 2019-01-04 - Tags: Thoughts - Description: The very first post on Web of Mike. - Word Count: 32 Well… I finally got around to creating the “Web of Mike”! I fully expect highly random articles around code, music, life, and whatever crazy idea comes to mind. So here we go! --- ## Best Practices for Securing Your APIs - URL: https://webofmike.com/best-practices-for-securing-your-apis/ - Markdown source: https://webofmike.com/best-practices-for-securing-your-apis/index.md - Author: Mike Moore - Published: 2019-01-04 - Last modified: 2019-01-04 - Tags: API, Tutorials - Description: A comprehensive look at API security covering encryption, OAuth, OpenID Connect, and more. - Word Count: 183 This presentation covers best practices for securing your APIs, including everything from encryption, to UUIDs, the differences between authentication and authorization, OAuth and OpenID Connect, and a host of other information around SSL, TLS, and more ways you can secure your APIs from those pesky would-be hackers. Much has been said around securing APIs and fortunately people do try to implement some of these practices. Unfortunately, most do not implement multiple legged security thereby relying solely on a single measure or two simplistic and easy-to-defeat mechanisms to secure their APIs. Alas, some even rely on not publishing the documentation for their APIs as a measure of “security”. These methods are simply not acceptable and it’s important to approach security by believing that “there are indeed smarter hackers than you out there and that your information is always at risk.” It is critical that you pay careful attention to security not as an afterthought, but as a well-formed strategy as you build your interfaces. Security should be engrained in every resource, every action, and truly everything you do as you build and design APIs! --- ## API Modeling and Design with Hypermedia and Contextual Information - URL: https://webofmike.com/api-modeling-and-design/ - Markdown source: https://webofmike.com/api-modeling-and-design/index.md - Author: Mike Moore - Published: 2019-01-04 - Last modified: 2019-01-04 - Tags: API, Thoughts - Description: Exploring hypermedia approaches to API design including JSON-LD, HAL, and Collection+JSON. - Word Count: 81 This presentation explores adding contextual and actionable information to APIs through Hypermedia types including JSON-LD, HAL, and Collection+JSON specifications. These standards enable richer API responses with embedded context and navigation capabilities. The core focus is on how hypermedia approaches can make APIs more self-descriptive and navigable, allowing clients to discover available actions and related resources dynamically rather than relying on out-of-band documentation. If you have any questions on anything API—be it development, management, design, or security, don’t hesitate to reach out. --- ## A Microservices Primer - URL: https://webofmike.com/microservices-primer/ - Markdown source: https://webofmike.com/microservices-primer/index.md - Author: Mike Moore - Published: 2019-01-04 - Last modified: 2019-01-04 - Tags: Microservices, API, Tutorials - Description: An introduction to microservices architecture and how it differs from monolithic and SOA approaches. - Word Count: 215 Microservices represent a modern architectural approach distinct from traditional monolithic systems. Rather than building applications as single, interconnected code bases, organizations divide functionality into smaller, independent services. The Current State of Things Many organizations currently operate using either Service-Oriented Architecture (SOA) or monolithic application designs. These approaches create fragility—a single bug can cascade throughout the entire system. Microservices offer an alternative path forward. Digital Transformation and Microservices The term “digital transformation” gains real meaning when organizations embrace microservices alongside containers, service discovery, and agile development methodologies. This approach breaks down monolithic systems into smaller, logically distinct functions typically aligned with organizational teams. The loosely coupled nature of microservices enables organizations to improve and deploy specific functions independently, avoiding full system regression testing with each deployment. Microservice Architecture A microservices architecture features several defining characteristics: Services remain easily replaceable Organization aligns with specific capabilities (user interfaces, billing, logistics) Implementation flexibility allows different programming languages and databases per service Symmetrical rather than hierarchical producer-consumer relationships This model supports continuous delivery and differs fundamentally from SOA by targeting single applications rather than integrating multiple business applications. APIs and Microservices API gateways provide consistent access layers across microservices, enabling combinations that deliver secure, flexible interfaces to end users while managing security, standardization, and performance optimization at the gateway level. --- ## Brew Day: Black IPA - URL: https://webofmike.com/brew-day-black-ipa/ - Markdown source: https://webofmike.com/brew-day-black-ipa/index.md - Author: Mike Moore - Published: 2015-02-15 - Last modified: 2015-02-15 - Tags: Thoughts - Description: A step-by-step brew day walkthrough making a Black IPA from a Northern Brewer Small Batch kit. - Word Count: 698 So it’s here again! BREW DAY. That glorious few hours where I get to drink beer while making beer. There’s really something chicken and egg about the whole thing. Which came first? Beer or beer? But I digress for now. As this is still somewhat of the beginning to my brewing adventures, I’ve been starting with some small batch 1-gallon kits to get a feel for where I’m headed next. Ideally I’ll be migrating onto some 5-10 gallon batches over the next year or so, getting some automation involved, and preparing a sweet taproom setup. So with all of that out of the way, let’s look at the most recent brew day where I put together a Black IPA. I used the Northern Brewer Small Batch Black IPA kit for this round and this was immensely better than getting started with the all-grain method that I started with during my first Beer Brewing Adventures. Black IPA Ingredients Steeping grains and mesh bag Gold malt extract syrup 7 grams Centennial hops 3.5 grams Chinook hops 3.5 grams Centennial hops 3.5 grams Cascade hops 7 grams Centennial hops 7 grams Cascade hops 5 oz Priming Sugar Safale US-05 brewer’s yeast Step 0: Sanitize the Equipment Sanitize above all else. Clean hands, equipment, surfaces, mind-set == great beer. This is the absolutely first and foremost thing you need to get in your head. If you don’t like to clean, maybe this sport isn’t for you. I use Star San as it’s a great wash that you don’t need to rinse out. You only need an ounce for 5 gallons of water so you can really get a lot of use out of this stuff. Step 1: Brewing the Wort First, you’ll want some decent quality water. You’ll need to put about 1.25 gallons (20 cups) of water in the kettle and start to bring it to a boil. While the kettle is heating up, go ahead and steep your grains in the water. Dump the grains in the mesh bag, and tie it off. Think of this as making tea with barley malt—steep for approximately 10 minutes as the water heats, then remove the grain and discard. Step 2: Boiling the Wort Once your wort comes to a solid rolling boil, start the timer at 45 minutes. Follow this exact addition schedule: 0 minutes in: Add Gold Malt Extract and 7g of Centennial hops 30 minutes in: Add 3.5g of Chinook hops 35 minutes in: Add 3.5g of Centennial hops 40 minutes in: Add 3.5g of Cascade hops 45 minutes in: Turn off heat, add remaining 7g of Centennial and 7g Cascade hops. Mix in the 5oz of priming sugar after the boil has stopped. Step 3: Chilling the Wort Ideally you’ll want to chill your wort as quickly as possible. The budget way to do this is in a sink full of ice and water. I use between 10lbs and 20lbs of ice and fill the sink with water. Stirring the ice around will help take the heat away from the kettle. You’ll want to do this until the kettle is cool to the touch. 60-70 degrees is ideal as this is where our yeast will be happy. Step 4: Transferring the Wort to the Fermentor Transfer your wort to the fermentor for fermenting. Make sure you’ve sanitized your gear! Transfer a gallon into your fermenter. Dump HALF and only half of the yeast into the fermentor. Place the cap on top, give the fermentor a healthy shake. Put the tube in, fill a small glass with water and a drop of Star San, and put the other end of the tube in the glass to let out CO2 but not let any contamination in. Step 5: The Long Wait! Put your fermentor setup in a cool (68-70 degree) dark place and wait. You’ll want to wait about 2-3 weeks or until the ferment is done. The first 24-48 hours, fermentation begins with a lot of foam and CO2. I’ll be doing a second ferment to clean up the beer for this round, so stay tuned for my next post. For now… I’ll wait impatiently for my tasty tasty beer… --- ## Beer Brewing Adventures - URL: https://webofmike.com/beer-brewing-adventures/ - Markdown source: https://webofmike.com/beer-brewing-adventures/index.md - Author: Mike Moore - Published: 2015-02-11 - Last modified: 2015-02-11 - Tags: Thoughts - Description: The beginning of Mike's home brewing journey. - Word Count: 107 As I begin expanding my website, I want to share my initial experiences with home brewing. This new hobby combines three of my passions: coding, automation, and beer consumption. My first batch is a small 1-gallon pale ale from a kit gifted by my wife, nearing completion. My inaugural attempt involved poor straining techniques, an all-grain approach, and unsuccessful siphoning efforts that resulted in about 9 bottles achieved out of the whole effort. I plan to pursue an extract-based Black IPA as my next project while continuing to refine my brewing skills alongside my primary focus on coding and software development. Follow my brewing journey right here! --- ## WordPress Optimization: Setting Up a Web Server - URL: https://webofmike.com/wordpress-optimization-setting-web-server/ - Markdown source: https://webofmike.com/wordpress-optimization-setting-web-server/index.md - Author: Mike Moore - Published: 2015-02-03 - Last modified: 2015-02-03 - Tags: Tutorials, Code - Description: Part 3 of the WordPress optimization series covering nginx, PHP-FPM installation and configuration. - Word Count: 148 Welcome to part 3 of an 8-part series on WordPress optimization focusing on web server configuration. This guide assumes completion of the previous installation steps for virtual server setup. Why Nginx? We’re in this for performance and nginx is an open source web server written to address some of the performance and scalability issues associated with Apache. Installation Steps Nginx Installation Basic installation requires one command to deploy nginx with extras. PHP-FPM Setup Installation and configuration of PHP-FPM for handling PHP execution, including pool settings for performance optimization with specific parameters for child processes and request handling. PHP Modules Install additional PHP modules including CLI, dev tools, XML-RPC, curl, GD, APC, and others for WordPress functionality. Configuration Files Three configuration files are needed: Common configuration for standard settings WordPress-specific rewrite rules Core nginx configuration Website Configuration Final step involves creating site-specific configuration files in /etc/nginx/sites-enabled/ with domain-specific settings. --- ## Kick-Ass WordPress Optimization - URL: https://webofmike.com/kick-ass-wordpress-optimization/ - Markdown source: https://webofmike.com/kick-ass-wordpress-optimization/index.md - Author: Mike Moore - Published: 2015-02-03 - Last modified: 2015-02-03 - Tags: Tutorials, Code - Description: An introductory roadmap for a comprehensive WordPress optimization series covering server setup through caching. - Word Count: 130 I’ve managed multiple WordPress sites with substantial traffic (up to 60K daily visits) on a single server. This article serves as an introductory roadmap for a multi-part series covering complete WordPress setup and optimization. I promise to break down complex server management into digestible sections for average bloggers, enabling them to operate independently from hosting providers. The guide encompasses eight major topics: Getting Started essentials Initial server setup procedures Web server configuration (nginx and PHP) CloudFlare DNS provider setup WordPress installation and configuration Website caching strategies Server performance tuning Search engine integration Each section addresses specific components, from domain registration through security hardening, to advanced caching systems using Varnish and Redis, and memory optimization across multiple services. Follow the upcoming detailed posts and transform your WordPress installation into a lean-mean-blogging-machine. --- ## WordPress Optimization: Setting Up A Virtual Server - URL: https://webofmike.com/wordpress-optimization-setting-server/ - Markdown source: https://webofmike.com/wordpress-optimization-setting-server/index.md - Author: Mike Moore - Published: 2015-02-01 - Last modified: 2015-02-01 - Tags: Tutorials, Code - Description: Part 2 of the WordPress optimization series covering virtual server setup on Digital Ocean. - Word Count: 284 This is Part 2 of an 8-part series on WordPress optimization, focusing on setting up your first virtual server. Why Anyone Can Do This IT professionals unnecessarily mystify technology. With proper explanation, anyone regardless of background can learn to set up and manage a server. Understanding Virtual Servers A virtual server exists in the cloud as a packaged set of computing resources—processor, RAM, and storage. This tutorial uses Digital Ocean as the preferred provider, referring to their servers as “Droplets.” These can be created in approximately 55 seconds. Creating Your Droplet The setup process involves selecting: Hostname: Your domain name Server Size: Recommendation starts at $5/month (1 CPU, 512MB RAM, 20GB disk) Location: Geographically close to your user base Additional Options: Private Networking, IPv6, and Backups Server Operating Systems Linux distributions are recommended over Windows for web servers. The tutorial specifically recommends Ubuntu 14.04 x64 for its long-term support status and ease of use. Accessing Your Server Mac users access servers through Terminal using SSH commands. Windows users use PuTTY, a standalone SSH client. Essential Linux Commands Command Purpose pwd Show current directory cd Change directories ls List directory contents passwd Change user password reboot Restart the server Security Configuration Creating a Non-Root User Never use the root account for daily operations. Create a new user with: useradd -g sudo -m -b /home/ [username] passwd [username] Firewall Setup with iptables Configure basic firewall rules to allow SSH (port 22), web server traffic (port 80), and established connections: sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT After completing these steps, you have a functioning and secured virtual server ready for web server installation. --- ## WordPress Optimization: Getting Started - URL: https://webofmike.com/wordpress-optimization-getting-started/ - Markdown source: https://webofmike.com/wordpress-optimization-getting-started/index.md - Author: Mike Moore - Published: 2015-02-01 - Last modified: 2015-02-01 - Tags: Tutorials, Code - Description: Part 1 of the WordPress optimization series covering essential prerequisites. - Word Count: 170 Welcome to Part 1 of a comprehensive 8-part series on WordPress optimization! This introductory chapter covers the essential prerequisites for getting started. Domain Name Before beginning, you’ll need to secure a domain name. Select one that reflects your blog’s subject matter, ideally incorporating your primary keyword. As long as you are genuine, follow the steps here, and give love and care to your content, the people will come! A Server You Can Work With I strongly advise moving away from wordpress.com hosted blogs to gain access to advanced optimization techniques. Acquire your own server through a provider like Digital Ocean, which offers affordable hosting solutions. You’ll need a server with command-line access to implement the performance enhancements covered in subsequent chapters. A Sense of Humor Server maintenance requires patience and persistence. This can be frustrating but following this guided tutorial will lead to success without unnecessary complications. Next Steps Once you have these three components in place, you’re ready to proceed to the next chapter: Setting Up Your First Server. --- ## Redirecting Bad Referers with Varnish - URL: https://webofmike.com/redirecting-bad-referers-with-varnish/ - Markdown source: https://webofmike.com/redirecting-bad-referers-with-varnish/index.md - Author: Mike Moore - Published: 2015-01-16 - Last modified: 2015-01-16 - Tags: Code, Tutorials - Description: A Varnish Cache VCL configuration for filtering and redirecting unwanted referral traffic. - Word Count: 113 Spam referrals cluttering analytics is frustrating. Here’s a VCL 4.0 approach to redirect suspicious referral sources. Implementation The solution uses a custom subroutine to identify and redirect problematic referrers: sub bad_referrals { if ( req.http.referer ~ "hulfingtonpost" || req.http.referer ~ "forum.topic57969834.darodar" || req.http.referer ~ "ilovevitaly" || req.http.referer ~ "priceg" || req.http.referer ~ "blackhatworth.com" ) { set req.http.host = "www.ihateyousomuchomgroflcats.com"; return(synth(750, "All your referer are belong to us.")); } } The configuration integrates into the receive and synthesis subroutines, ultimately directing flagged traffic via HTTP 301 redirect to an alternative destination. Administrators can customize the referrer matching patterns and redirect targets according to their specific needs, effectively removing spam referral sources from analytics reporting. --- ## Socket.io Cluster with Nginx as Reverse Proxy - URL: https://webofmike.com/socket-io-cluster-with-nginx/ - Markdown source: https://webofmike.com/socket-io-cluster-with-nginx/index.md - Author: Mike Moore - Published: 2015-01-08 - Last modified: 2015-01-08 - Tags: Code, Tutorials - Description: Using Node.js cluster module with websockets and nginx as a reverse proxy. - Word Count: 62 Node.js by default runs on a single process and at max utilizes one CPU. To take the full advantage of a multi core system, multiple node processes can be run with a frontend proxy interfacing with the client. This article discusses using Node.js’s cluster module with websockets and nginx as a reverse proxy to achieve better performance and scalability for real-time applications. --- ## Deploying WordPress over Nginx and PHP-FPM - URL: https://webofmike.com/deploying-wordpress-over-nginx-and-php-fpm/ - Markdown source: https://webofmike.com/deploying-wordpress-over-nginx-and-php-fpm/index.md - Author: Mike Moore - Published: 2015-01-08 - Last modified: 2015-01-08 - Tags: Tutorials, Code - Description: Resources for advanced WordPress deployment using Nginx and PHP-FPM. - Word Count: 78 This post highlights advanced deployment resources for setting up WordPress infrastructure from the ground up. Key installation components include: Installing Nginx from package repository or compiling it from scratch PHP5, php-mysql, php-fpm, and additional PHP libraries (such as php-gd) MySQL or MariaDB database systems Configuration of PHP, Nginx, and database services For those new to WordPress administration, check out the Kick-Ass WordPress Optimization Series on this blog, which provides step-by-step guidance covering virtual server creation through Nginx/PHP-FPM optimization. --- ## Tornado Error Handling - URL: https://webofmike.com/tornado-error-handling/ - Markdown source: https://webofmike.com/tornado-error-handling/index.md - Author: Mike Moore - Published: 2014-02-09 - Last modified: 2014-02-09 - Tags: Code, Tutorials - Description: Implementing custom error handling in the Tornado web framework using a reusable base handler class. - Word Count: 126 This post discusses implementing custom error handling in the Tornado web framework by creating a reusable base handler class. The Problem I identified a need for custom error messages and a more robust foundation for request handlers across my Tornado application. The Solution The approach involves creating a base handler class that all application handlers inherit from. The key implementation uses the write_error method to intercept HTTP errors: class BaseHandler(tornado.web.RequestHandler): def __init__(self, application, request, **kwargs): super(BaseHandler, self).__init__(application, request) def write_error(self, status_code, **kwargs): if status_code == 404: self.render('errors/404.html', page=None) else: self.render('errors/unknown.html', page=None) Implementation Handlers then inherit from this base class: class MainHandler(BaseHandler): def put(self): pass def get(self): pass def post(self): pass def delete(self): pass This pattern enables centralized error handling for all CRUD operations across the application. --- ## Tornado Sessions with Redis - URL: https://webofmike.com/tornado-sessions-with-redis/ - Markdown source: https://webofmike.com/tornado-sessions-with-redis/index.md - Author: Mike Moore - Published: 2014-02-07 - Last modified: 2014-02-07 - Tags: Code, Tutorials - Description: Implementing session management in Tornado web framework using Redis as the session store. - Word Count: 152 Tornado really doesn’t have a great native session handler, so I created a custom solution combining Tornado with Redis for session management. Session Storage Implementation The RedisSessionStore class manages backend operations: Initializes with configurable key prefixes and expiration times (default: 60 days) Generates session IDs using UUID Stores/retrieves pickled session data in Redis hash structures Automatically expires sessions based on configuration Session Wrapper Class The Session class provides a dictionary-like interface: Lazy-loads session data from Redis Tracks modification state via a “dirty” flag Implements standard Python container methods (__getitem__, __setitem__, etc.) Records access timestamps with IP addresses Auto-saves changes upon deletion Tornado Integration class BaseHandler(tornado.web.RequestHandler): def get_current_user(self): return self.session['user'] if self.session and 'user' in self.session else None @property def session(self): sessionid = self.get_secure_cookie('AUTH_COOKIE', None) if sessionid: return Session(self.application.session_store, sessionid) else: sess = Session(self.application.session_store, None) self.set_secure_cookie('AUTH_COOKIE', sess.sessionid) return sess The implementation works with local Redis instances but supports remote connections through customized parameters.