Contents (7 sections)
On paper, Proxmox high availability is a box to tick: you add a resource to HA, and if a node dies, it restarts elsewhere. We run a three-node cluster with 31 HA resources (everything that can be), and the road between the ticked box and a cluster that actually behaves as intended is paved with discoveries. Here they are, to spare you them.
First, what HA does (and does not do) #
Proxmox HA restarts your machines on another node when theirs becomes unreachable. Restarts: so there is an interruption, and on replicated local ZFS storage (our case, no shared SAN), the machine starts again from the last replica. With replication every 15 minutes, your real RPO is 15 minutes: that is the first figure to know, and to compare with what your disaster recovery plan requires. HA protects against the hardware failure of a node, not against ransomware or human error, which replicate dutifully to the other node in under 15 minutes.
A structural corollary on local ZFS: a machine without a replica has no business being in HA, since failover would start it on a node that does not have its disk. On our side, the rule is mechanical: the allowed nodes of an HA resource are derived from its replication, never from guesswork. And a prerequisite before enabling anything at all: a redundant cluster network (dual corosync ring), otherwise a simple network loss turns into fencing, with the cluster “killing” a healthy node out of excessive caution.
The myth: no, containers do not live-migrate #
It is written in the Proxmox code, not in the brochures: an LXC container cannot live-migrate. During maintenance as during failover, a CT is stopped, synchronised, restarted: count on 10 to 30 seconds of interruption, plus the application start-up. VMs, on the other hand, live-migrate, local disks included. This detail changes sizing: a service that tolerates no blip needs a VM (or application-level redundancy), not a CT “in HA”. We checked it both ways: our CTs do their 10 to 30 second relocate, and a notoriously temperamental VM live-migrated without losing a second of uptime.
The three pitfalls that cost us hours #
1. Without failback, machines never come back #
The shutdown_policy=migrate policy cleanly evacuates resources when you reboot a node. What we learned the hard way: the return depends on each resource’s failback flag. With failback 0 (the historical default in our configuration), a node reboot ended with nine machines that never came back: the node empty, its neighbour overloaded, and an HA table calmly showing started everywhere. The drift is silent and cumulative: each maintenance unbalances the cluster a little more. Since then, failback 1 everywhere: each node return costs a few 10 to 30 second relocates, a price we accept in exchange for no invisible drift.
2. The HIGHEST priority wins #
In a placement rule (node-affinity), nodeA:1, nodeB:2 means that nodeB is the preferred node: the highest number wins. Read intuitively the wrong way round, you set exactly the opposite of your intention, and HA moves the machine within the next minute… which, for a container, means an immediate stop and restart. We fell into this trap twice in one month, hence this line in our procedures: always check which number is highest before validating a rule.
3. qm stop does not stop an HA resource #
A resource under HA belongs to the HA manager: stop it by hand and the manager considers it failed… and therefore restarts it. The day a VM freezes during a backup (it happened to us, thanks to the interaction between file system freezing and security software in the guest), the qm stop reflex is no longer enough: it is ha-manager crm-command stop that stops it cleanly, informing the manager. Something to know before the incident, not during it.
What we take away #
- HA is designed from replication: RPO of 15 minutes, allowed nodes = replicated nodes, capacity of the surviving node checked (losing a node concentrates its load on its replica neighbour; the failover map is drawn while things are calm).
- CT = restart, VM = live migration: choose the right vehicle for each service.
failback, priorities, stopping an HA resource: three counter-intuitive behaviours to test on a machine that does not matter, never to discover in production.- And in all honesty: failover on a hard node failure (unplugging it, not rebooting it) remains an exercise we still have to schedule. An untested HA cluster is a hypothesis, not a guarantee.
This infrastructure carries our sovereign hosting: automatic failover, replication, immutable backups and monitoring. If your hosting provider tells you “it is in HA” without being able to answer on RPO or failback, you now know which questions to ask, and so do we.
