incident log / 3 entries
Production incidents written up honestly: the problem, the investigation, the root cause, and what I'd do differently. No sanitized success stories — the value is in the dead ends.
Backend views started throwing NoneType errors on Redis reads, and the Redis pod's logs were spotless — because OOMKill is a SIGKILL, and dead processes don't write farewell notes.
Redis · Kubernetes · Django—3 min read
Zero snapshot-cleanup jobs plus 200% over-provisioning let dead Longhorn snapshots pin ~100GB of ghost blocks under the Harbor registry volume. It filled a 196GB disk, the filesystem went read-only, and every build in the cluster started failing with I/O errors.
Longhorn · Harbor · Kaniko · ext4—4 min read
Seven workers hit disk pressure, and every standard fix made it worse: restarted agents flapped back NotReady in minutes, kubelets hung silently for 30+ minutes on startup, and Harbor — the registry every recovering pod needed — went down with the cluster it runs on.
RKE2 · Longhorn · Harbor · Argo CD · kubelet—4 min read