A multi-cluster Kubernetes platform for AI agents (lab)
Five Flux-managed clusters, 30+ Helm releases, and a custom CRD plus TypeScript control plane that provisions gVisor-isolated machines for AI agents. 1,000+ commits since August 2026, and the test track for the infrastructure I want to build next.
Context
Production orchestration at Tembo Health was ECS and Fargate. Kubernetes is where I think the next generation of AI agent infrastructure gets built, so in August 2026 I started a lab to learn it the way I learn best: by running something real, with GitOps discipline, and breaking it. It is a lab, not production, and I say so on my resume. It is also 1,000+ commits of working platform.
What runs
Five Flux-managed Kubernetes clusters: k3s on an Oracle Cloud VM, Talos on lab hardware provisioned through Sidero Omni, a staging vCluster, a one-node upgrade-rehearsal cluster, and a dedicated data cluster. Cilium CNI, Traefik, cert-manager, external-dns, Dex OIDC and 30+ Helm releases. Secrets come from Bitwarden through External Secrets Operator and never touch git, so any cluster rebuilds from the repo plus one token.
Observability, data and CI: kube-prometheus-stack with Prometheus agent mode remote-writing to a central Prometheus, Grafana with alerting, replicated CloudNativePG Postgres with off-site backups, Forgejo workflows (GitHub Actions compatible) that build and push the platform’s own CI runner and agent machine images to Forgejo’s registry, and Renovate for dependency updates.
The control plane
The point of the lab is agents. I designed a custom Kubernetes CRD and a TypeScript control plane (Effect, Hono, Metacontroller) that provisions gVisor-isolated AI agent machines with a desktop, terminal, browser and pluggable CLI harnesses, Cilium default-deny networking and per-tenant quotas. The API adds a Postgres-backed job queue, an event ledger with signed webhook deliveries and replay, OIDC sessions, workspaces and API keys, and an OpenAPI spec.
Over that API sits an operator console in React (TanStack Start, Router, Query and Form, shadcn): machine pages, an embedded noVNC desktop and terminal, a command palette, workspace and API-key settings, and a Scalar API reference. 27K+ lines of TypeScript with 47 test files across API and console.
Why it matters
This is the infrastructure question I find most interesting right now: what it takes to run autonomous agents at scale, each with its own isolated machine, identity and network policy. The lab is where I work that out. The repo is private for now.
Stack
Kubernetes · Flux · Talos · k3s · Cilium · Helm · CloudNativePG · Prometheus · Grafana · Forgejo · gVisor · TypeScript · Effect · Hono · Metacontroller · React · TanStack