Healthy, Attached, and Read-Only: When Your Storage Control Plane Lies

The ticket said the vector database was down. The pod was Running. The Longhorn dashboard showed its volume attached and healthy — green across the board. And the application was getting EIO on every single write.

One of these witnesses had to be lying. The uncomfortable answer: none of them were.

Two layers, two truths

When a distributed block store like Longhorn reports a volume as healthy, it's making a claim about its own layer: the replica processes are running, they're consistent with each other, the block device is attached to the node and serving reads and writes at the block level. Every word of that can be true.

But inside that block device lives a filesystem, and the filesystem has opinions of its own. During an earlier disk-full event, this volume's ext4 had hit a write failure serious enough to abort its journal. And ext4's default response to journal abort is exactly what its mount option says:

$ dmesg | grep -i ext4
EXT4-fs error (device sda): ext4_journal_check_start:83: Detected aborted journal
EXT4-fs (sda): Remounting filesystem read-only

errors=remount-ro. The filesystem protects itself from further corruption by refusing all writes — silently, from the perspective of everything above and below it. The block device keeps serving I/O (Longhorn: healthy ✓). The kernel keeps the mount alive (pod: Running ✓). And every write returns EIO (application: down ✗).

The control plane wasn't lying. It was telling the truth about the layer it owns — which is one level below the layer that broke.

Seeing past the dashboard

The gap generalizes: a storage control plane attests to the health of what it manages, not what you stored in it. Checks that cut through, from cheapest to most thorough:

1. dmesg on the node — filesystem errors, journal aborts, and remount events are all there, timestamped. This is the single highest-value stop.
2. Check the actual mount flags: grep ' ro,' /proc/mounts from the pod's mount namespace (or a touch test in the volume path). A read-only remount is invisible in most dashboards and one command away in the kernel's own records.
3. Distrust "Running" as a health claim. The pod's process may be alive and failing every operation. Liveness probes that only prove the process exists — rather than exercising a write path — will happily keep a broken pod "healthy" too.

The recovery that preserves data

The tempting move is delete-and-restore. The right move, when the damage is a journal abort rather than physical corruption, usually preserves everything:

1. Scale the workload to zero so nothing holds the mount.
2. Maintenance-attach the volume — Longhorn (and most CSI storage systems) can attach a volume to a node without a workload, exposing the raw block device.
3. Run e2fsck against the raw device. For a journal abort, this typically replays or discards the journal and repairs metadata — minutes of work.
4. Detach, scale the workload back up, verify with an actual write.

In our case: full recovery, data intact, no restore from backup needed. The filesystem had done its job — remounting read-only is the data-preservation mechanism. It stopped the damage at the moment of the fault and waited for someone to come fix it properly.

The takeaway

"Healthy" always has an implied scope: the layer the reporting system owns. Longhorn's green checkmark covers replicas and block devices. The kubelet's Running covers the process. Neither covers the ext4 journal, and nothing above the kernel covers it at all — except dmesg and your own write test.

When the application and the infrastructure dashboards disagree, don't pick a side. Find the layer between them that neither one is watching. That's where these incidents live.