journal / 3 entries
Engineering notes worth keeping: things I learned in production, explained the way I wish someone had explained them to me.
A CORS failure that was a file permission. A registry auth error that was a full disk. In layered systems the error tells you where the failure surfaced, not where it lives — four production examples and the habit that cuts through them.
OOMKill is a SIGKILL: the process gets no chance to log, so the pod's logs look innocent while the whole fleet throws NoneType errors. Where the evidence actually lives, and the write-time habits that prevent the class.
Longhorn said the volume was healthy. The pod got EIO on every write. Both were right — about different layers. What control-plane health actually attests, and the e2fsck recovery that preserves data.