Reliability

Disaster recovery plan: what it really contains

Everyone has “backups”. Almost nobody knows how long it would take to restart, in what order, or who decides. RTO, RPO, written procedures, drills: the real content of a disaster recovery plan at SME scale, without consultancy jargon, with what we run in production.

Falling servers stopped by a barrier, the infrastructure restarting as a mirror image on the other side
Contents (4 sections)
  1. RTO and RPO: the only two acronyms that matter
  2. What an SME disaster recovery plan really contains
  3. The technical means, layer by layer
  4. A plan that is never rehearsed is fiction

Ask around: “if your main server burned down tonight, how long before the company was working again?”. You will get silence, or rough guesses ranging from “a few hours” to “no idea”. That is exactly the gap a disaster recovery plan fills: turning “we have backups” into “we restart in N hours, losing at worst M minutes of data, and we have done it before”.

RTO and RPO: the only two acronyms that matter #

A disaster recovery plan rests on two figures, and they are business decisions, not technical ones. The RTO (Recovery Time Objective): how long can the business keep going without the service? The RPO (Recovery Point Objective): how much data can you afford to lose: the last few minutes, the last hour, the last day? An accounting firm can live half a day without its tool but cannot lose a single entry; an online shop is the opposite. Until management has set these two figures, all the technology underneath is tuning in the dark.

Without a decided RTO, you still have one: the one you will discover on the day of the outage. It is usually longer than the business can bear.

What an SME disaster recovery plan really contains #

Forget the 80-page binder gathering dust in a cupboard. A useful disaster recovery plan is short, up to date, and above all executable by someone tired at 3 am:

  • An inventory by criticality: which services, which machines, which dependencies. Without it, you restore in the wrong order: the application before its database, the database before DNS.
  • RTO and RPO per service: not everything is equal. E-mail, invoicing and the intranet have neither the same urgency nor the same need for fresh data.
  • Written restore procedures: not “restore the server”, but the commands, the access details, the checkpoints. If the procedure only exists in one person’s head, your plan has a human single point of failure.
  • The restart order: network, DNS, databases, applications, then client machines. Dependencies dictate a sequence, so write it down while things are calm.
  • Roles and contacts: who decides to trigger the plan, who restores, who informs customers. With phone numbers, not “see the directory”, which may well be on the failed server.
  • Trigger criteria: at what point do you switch from troubleshooting to recovery? The worst losses come from hours spent repairing something already dead.

The technical means, layer by layer #

The plan is the document; you still need the means to make it workable. They stack up in layers, from cheapest to most demanding, and each layer reduces RTO or RPO:

  1. Tested, immutable, offsite backups: the absolute foundation, without which the rest is decoration. We described this architecture in detail in our 3-2-1 article. Typical RPO: a day or an hour; RTO: the time it takes to restore.
  2. Replication: data copied continuously or every X minutes to a second host. RPO drops to a few minutes; RTO remains that of a failover.
  3. High availability: several nodes, automatic failover when a host fails. RTO drops to a few minutes, for hardware failures, not for ransomware or human error, which replicate as fast as the data.
  4. Independent monitoring: who alerts you if the outage happens at 2 am? And what if the monitoring itself dies? An external “dead man’s switch” check (a probe that worries about the silence of the monitoring system) closes the loop.

This is the stack we run for our clients: a multi-node cluster with automatic failover, replication between hosts, daily immutable backups with an offsite copy, and monitoring backed up by an independent watchdog. None of these components is exotic; what is rare is having all of them in place and verified.

A plan that is never rehearsed is fiction #

The day of the outage is not the right day to discover that a password has changed, that a procedure is two years out of date or that a restore takes six hours, not two. One drill a year, at the very least: actually restore a critical service on a blank host, time it, correct the procedure. Our first full drill taught us details that no document review would have caught: that is precisely what it is for. The stopwatch of the drill is your real RTO; compare it with the one management decided, and you know where to invest.

Our managed services include this foundation (tested backups, monitoring day and night, on-call support) and we build the disaster recovery plan with you: inventory, RTO and RPO set with management, written procedures, annual drill. If you cannot answer the question in the first paragraph, now is the right time to talk about it.

About
01/Read nextThree articles

Related articles

Infrastructure

Exploit foundries: your firewall has become the front line

Two undergraduates in Changsha ran an automated pipeline that decrypts firmware, decompiles it through thousands of calls, forms hypotheses about vulnerabilities, writes the exploit and tests it in a lab, in a loop, day and night. More than a dozen unknown vulnerabilities in a month on network devices, some fifty organisations targeted, thirteen collection agents on a scheduled task. What this means for an SME's inventory.

cybersecurityvulnerability
Infrastructure

Intermittent SERVFAIL: the orphaned DS record that broke 15% of lookups

A CI pipeline failing intermittently, two different error messages, an obvious suspect that had nothing to do with it, and an invisible cause: a DNSSEC record still published at the parent while the zone was no longer signed. Post-mortem of a nasty DNS failure, with the method that finally settled it.

dnssecdns
Infrastructure

Patching 36 servers without unattended-upgrades

There are three doctrines: do nothing, automate everything blindly, or look first and then decide. We removed unattended-upgrades from the whole fleet and dismantled our own automatic patching with Ansible, in favour of a “visibility first” doctrine backed by PatchMon. Why, how, and the routine that goes with it.

patch managementpatchmon

What next

Infrastructure that is managed and monitored day and night?

Bastion, SIEM, patches applied on purpose, tested backups: what we apply for our clients. First conversation free.

Discuss your project

Reply within 24 working hours, from one of the two founders. Or by phone: +33 7 59 96 14 99.