Contents (4 sections)
When it comes to security updates, a server fleet has only three possible doctrines. Do nothing, the most widespread and indefensible. Automate everything: unattended-upgrades everywhere, packages install themselves at night. Or look, then decide: continuous visibility on the state of each machine, and updates applied through a human decision backed by the right tools. We practised the second, then dismantled even our own home-grown automation to adopt the third. Here is why, and the complete setup.
The problem with unattended-upgrades is not what it does, it is what it does not see #
To be fair: on an isolated workstation or a personal server, unattended-upgrades is infinitely better than forgetting. Our objection applies to a production fleet: a patch is an untested change, applied at a time you do not choose, with no human to check the result. A service that restarts in the middle of the night and does not come back, a dependency that breaks an application, a fix requiring a full reboot left pending for weeks: blind automation turns every night into a deployment with nobody at the controls. And its failure is silent: the package that fails to install warns nobody, and the machine that is “up to date” in your mind drifts in reality. Across thirty machines with different roles, a random night-time outage costs more than the few days’ delay of a deliberate patch.
An intermediate step: home-grown automation, and why we removed it too #
Our first answer was centralised patching with Ansible: playbooks, a schedule, reports. Honest, but two structural flaws ended up weighing on it. First, no live inventory: between two runs, no idea of the fleet’s real state: how many fixes pending, where, since when. Second, the cron job patched without looking: the same blind automation, just written by us. The lesson: the problem was not the tool that applied the patches, it was the lack of permanent visibility between actions.
The current doctrine: PatchMon, look then decide #
Today, a lightweight agent on each of the 36 machines in the fleet reports continuously to a self-hosted (open-source) PatchMon instance: pending packages, security fixes, versions, history. The dashboard answers at any moment the question “where does the fleet stand?”, and patching becomes what it should always have been: an informed decision. From the same console, a patch run targets a machine or a group, at the chosen time, with its result recorded. More than eighty runs carried out to date, zero failures: not because the tool is magic, but because every run is done while watching.
The routine that makes patching boring (that is the point) #
- Look before: what is going to happen, which services are affected, will a service or machine restart be needed?
- Snapshot before any significant update: on a hypervisor, a machine snapshot costs a few seconds and turns the worst-case scenario into a five-minute rollback.
- Order of rollout: development environments and machines that do not matter first, production afterwards, never everything at once.
- Check after: the application health check: not “the package is installed” but “the service responds”. This is the step blind automation will never perform.
- Accepted exceptions: a major database version upgrade is a migration, not a patch: never automatic, never in a batch.
Visibility on the fleet, deliberate runs, snapshots before, checks after: that is the concrete content of our managed services. If your current answer is “unattended-upgrades and fingers crossed”, or worse “we will see later”, let us compare notes.
