Skip to content
nayem.talukder
← case studies

Running 25+ Ephemeral QA Environments on Self-Managed RKE2 — and Surviving the Storage

A rapid-QA tier of 28 namespaces on a self-managed cluster where Longhorn shared the root disk with kubelet and reserved nothing. One full disk became a cluster-wide eviction cascade; this is the reliability campaign that followed.

role: Platform Engineertimeframe: 2025 – 2026read: 3 min

stack: RKE2 · Longhorn · Argo CD · Tekton · Harbor · KEDA

Problem

Product teams needed fast, disposable, full-product QA environments — spin one up per feature stream, run realistic workloads against it, tear it down. The answer was a rapid-QA tier: 28 namespaces of complete product environments on a self-managed RKE2 cluster, with in-cluster CI and GitOps delivery.

The job of this tier is speed. The story of this case study is what it cost to keep it alive — and the honest question that emerged at the end about whether the architecture was right at all.

Architecture

  • Delivery: Argo CD syncs every environment from Git; Tekton pipelines build in-cluster with change detection, rebuilding only modified services and retagging the rest.
  • Registry: Harbor, running inside the same cluster it serves.
  • Autoscaling: KEDA scales document-processing workers on queue depth, so idle environments cost near nothing and busy ones keep up.
  • Storage: Longhorn distributed block storage — with the design flaw that shaped a quarter of my year: Longhorn's data path shared the root filesystem with the kubelet, with zero reservation separating them, 200% over-provisioning, and no snapshot cleanup configured.

That storage arrangement means storage pressure and node pressure are the same disk wearing two hats: either one filling triggers the other's failure mode.

Technology Choices

  • Self-managed RKE2 over managed Kubernetes: this tier ran on infrastructure we owned, where per-environment cost had to stay near zero. The trade is that every layer — from multipathd defaults in the VM template to iSCSI device claiming — is yours to operate.
  • In-cluster CI and registry: minimal external dependencies and fast image paths, at the price of circular dependencies during incidents (the registry going down with the cluster that needs it to recover).
  • Retag-instead-of-rebuild in CI: most commits touch a few services; copying manifests for unchanged ones cut build times dramatically — and created a race with registry garbage collection that had to be discovered the hard way.

Challenges

The storage design flaw expressed itself as a sequence of production incidents, each documented in full as war stories:

  • The May 12 cascade — seven nodes under disk pressure, an eviction storm, Harbor down, and every standard remediation making it worse. Four hours to stable; the centerpiece incident of the campaign.
  • Snapshot bloat — dead Longhorn snapshots with no garbage collector pinning ~100GB of ghost blocks under the registry volume until a 196GB disk hit 100% and went read-only.
  • Full-disk deadlocks — failed replicas holding their space by design while the auto-salvage waited for a schedulable disk that a full disk can never be.
  • Fleet-provisioning gaps — nodes joined without the label Longhorn needed, multipathd claiming iSCSI devices before kubelet could — template-level defects that fired weeks after the nodes joined.

Trade-offs

  • Speed over isolation, explicitly. Environments share nodes, storage, and the registry. That's why the tier is cheap and fast, and why one namespace's disk appetite could become everyone's incident — the reliability campaign existed to put guardrails around a sharing model we deliberately kept.
  • Remediation as automation, conservatively. Every recurring incident ended as a script, CronJob, or admission policy: ephemeral-storage LimitRanges on every tenant namespace, daily snapshot-cleanup RecurringJobs, weekly orphaned-sandbox cleanup, a storage-health CLI, and an agent-tunnel watchdog. The watchdog runs in dry-run — automation that can restart node agents earns trust before it acts alone.
  • Playbooks before automation. Nothing got a CronJob until it had been executed by hand at least twice and written down. Automating a procedure you don't fully understand converts an incident into a recurring mystery.

Results

  • The tier survived its worst quarter and came out with guardrails instead of luck: resource limits that bound the blast radius, garbage collection at every layer that needed it, and watchdogs for the failure modes that recur.
  • Incident response went from improvisation to playbooks — the class of 2am surprise that started the campaign now either can't happen (limits), cleans itself up (RecurringJobs), or gets flagged before users notice (watchdogs in dry-run, graduating as they prove out).
  • The honest close: after making it survivable, the open question is whether this tier should be Kubernetes at all. Production parity is gated downstream on the client's OpenShift regardless — so vendor-side Kubernetes buys orchestrator parity that rapid QA may not need, at a real operational tax. The follow-up work is a decision record weighing plain Linux hosts with docker-compose and the same images. A case study that ends with "and we might replace it" is, I think, the most useful kind.