The Crash With No Error Message

Problem

Backend views in an enterprise environment started intermittently throwing NoneType errors on Redis reads — values that should have existed simply weren't there. The obvious suspect was Redis, so we pulled the pod's logs.

They were spotless. Clean startup, no errors, no warnings. According to its own logs, Redis had never had a bad day in its life.

Context

The platform's Django backend used Redis for job state: document-processing payloads written by workers and read back by views. The instance ran in Kubernetes with a 1Gi memory limit. The workload had been growing for months — and, as it turned out, so had something else.

Investigation

The logs were clean because we were asking the wrong witness. kubectl describe pod told the real story:

Last State:  Terminated
  Reason:    OOMKilled
  Exit Code: 137
Restart Count: 13

An OOMKill is a SIGKILL from the kernel. The process gets no signal handler, no flush, no final log line — it just stops existing. The pod's current logs belong to the fresh process that replaced it, which naturally has nothing to complain about. The evidence of the crash lives in the pod's Last State, the restart count, and the node's kernel events — never in the application log.

With the right lens, the mechanism was easy to trace. Job payloads of 10–24MB each had been written without TTLs for months. Nothing ever expired; memory walked steadily toward the 1Gi limit until the kernel stepped in.

Then the loop that made it vicious: Redis persisted its dataset via RDB snapshots. Every OOMKill was followed by a restart that reloaded roughly 1GB of the same bloat from disk — and promptly got killed again. One environment had cycled through 13 restarts this way. Persistence, which exists to protect your data, was faithfully preserving the exact state that was killing the process.

Each kill also explained the NoneType errors: reads that raced the crash window found nothing, and the application code trusted the cache to always answer.

Root Cause

Three small decisions compounding: job keys written with no TTL, no eviction pressure-release configured for the workload's growth, and RDB persistence reloading the accumulated bloat on every restart. None of them was a bug on its own. Together they were a slow-motion crash loop with a months-long fuse.

Solution

Immediate: purge the stale job keys (careful SCAN-based deletion, not FLUSHDB — live state was interleaved with the dead), then BGSAVE so the cleaned state is what any future restart reloads. That broke the reload-and-die cycle on the two affected environments.

The real fix was at write time: every job payload now carries an explicit TTL (ex= at the point of write), so the working set is self-limiting instead of append-only.

Then the fleet check, which turned out to be the most valuable hour of the incident: the same bomb was ticking everywhere. Three more environments were sitting at 63–82% of their memory limit with the same no-TTL write pattern. They got the same remediation before they ever paged anyone.

Technical Decisions

• SCAN-based selective purge over FLUSHDB: flushing would have been faster and also would have destroyed live job state for in-flight work. Slow and surgical was right.
• TTL at write time over a bigger memory limit: raising the limit moves the cliff; it doesn't remove it. Unbounded keys plus finite memory always ends the same way — the only question is the date.
• Proactive fleet remediation: once a failure mode is understood, checking every other environment for it is cheap. Waiting for each one to fail independently is how one incident becomes four.

Lessons Learned

• Check Last State, not logs. SIGKILL leaves no trace in the application's own output. kubectl describe, restart counts, and exit code 137 are where OOM evidence lives.
• A clean log is not evidence of a healthy process — it can equally be evidence of a process that died too fast to say anything.
• Persistence preserves your mistakes with the same fidelity as your data. Any restart-based recovery has to account for what gets restored.
• Every cache key without a TTL is a small loan against future memory. The interest compounds quietly until the kernel calls it in.