Post-mortem of an NVMe failure: the dead capacitors that slowed everything down

Disk failures in films are binary: it works, then it dies. Real failures are more devious. This one degraded a production hypervisor for two and a half months without ever “breaking” anything: no disk out of service, no data lost, no red alert, just sporadic timeouts at night, and a component a few millimetres across that had stopped doing its job. A full post-mortem, because this failure mode lies in wait for any server.

PLP: the component nobody talks about #

Power Loss Protection is what separates an enterprise SSD from a consumer SSD. An SSD honours “safe” (sync) writes in one of two ways: either it really writes to NAND before replying (slow), or it replies from its DRAM and commits to finishing the job, a promise it can only keep in a power cut thanks to a reserve of on-board capacitors that give it the milliseconds needed to flush its cache. Those capacitors are the PLP. Without it, an SSD that wants to stay honest has to disable its sync write cache, and its performance collapses. For a file system like ZFS, which issues many synchronous writes, PLP is not a luxury: it is what the advertised latencies depend on.

The symptom: timeouts at night, and nothing else #

It all starts with ZFS replication failures: zfs snapshot … got timeout, sporadic, concentrated in a window of a handful of minutes each night, right when application backups were loading the guests with writes. Two hundred and forty-six failures piled up before the investigation traced them back to the source. The measurement that revealed everything: the write latency of the two NVMe drives in the mirror. The healthy disk responded in 52 microseconds; its twin in 4 to 6 milliseconds, a hundred times more. And a mirror moves at the pace of its slowest member: the whole pool was waiting.

smartctl -H /dev/nvme1
# SMART overall-health self-assessment test result: FAILED!
# - volatile memory backup device has failed

smartctl -a /dev/nvme1 | grep -i critical
# Critical Warning: 0x10   <- bit 4: volatile memory backup failure

# And yet the NAND was HEALTHY:
# Percentage Used: 5% · Media and Data Integrity Errors: 0

The SMART verdict fits in one line: volatile memory backup device has failed: the PLP capacitors were dead. The controller, being honest, had disabled its write cache and was absorbing every sync at NAND speed. The disk was neither dead nor dying: its NAND showed 5% wear and zero errors. A safety component had given out, and the only trace was a SMART line nobody reads… since the disk “works”. The date the fault first appeared in the logs: eleven weeks before the diagnosis. Lesson number one: monitor the SMART detail (the health status and the Critical Warnings), not just “the disk responds”.

The replacement: what really happens at a hosting provider #

On a dedicated server, you do not change the disk yourself: you open a ticket with the hosting provider (OVH in this case) with the disk’s serial number and the SMART report, you schedule a slot, and a data centre technician replaces the disk. On our side, bringing a mirror member back into service takes four steps, each with its own trap:

1. Copy the partition table from the surviving disk: sgdisk -R <destination> <source> then -G to regenerate the GUIDs. Classic trap: the argument order is destination first: swap them and you overwrite the healthy disk’s table.
2. Reintegrate each layer: the md partitions (mdadm --add), and the swap recreated with its original UUID (mkswap -U …) so that the existing configuration finds it at boot.
3. zpool replace on the ZFS partition: the resilver copied 218 GB in five minutes.
4. Check the enumeration: at the next reboot, /dev/nvme0 and /dev/nvme1 had swapped names. Anything that references a disk must do so by a stable identifier (/dev/disk/by-id/, serial number), never by enumeration number.

Result measured after replacement: 236 µs write latency, symmetrical across both disks, under nearly 4,000 IOPS. The night-time timeouts disappeared that same evening.

The final twist: 20,000 errors lying dormant #

The resilver had a valuable side effect: it was the first full read of the pool since it was created. And that read dug up more than twenty thousand checksum errors on cold data: old frozen replicas, not read for months, whose corruption had therefore never been seen. No production data was affected and cleaning up was enough; but the lesson is exactly the one in our article on scrubs, written here in letters of fire: data that is never read can be corrupted for months without a single warning light coming on. Since then, a monthly scrub is scheduled and verified on every node: the failure will at least have paid for its lesson.

This kind of hunt (correlating night-time timeouts, a backup window and a SMART line) is everyday work for our managed services: reliability is not the absence of failures, it is seeing them before they cost you. If your servers are more than two years old and nobody has ever read their SMART data in detail, now is the time.