Most disaster recovery plans are written to satisfy an auditor and have never been executed. The test of a plan is not whether it exists but whether someone who did not write it can follow it under pressure.
Start with two numbers
Recovery time objective (RTO) is how long the service can be down before the consequences become unacceptable.
Recovery point objective (RPO) is how much data you can afford to lose, measured in time. An RPO of one hour means losing up to an hour of writes is survivable.
These are business decisions, not technical ones, and they must be agreed with whoever owns the customer relationships and the contracts. Engineers asked to pick will usually choose numbers that are expensive to achieve; executives asked without context will say zero.
The useful framing is cost. An RPO of 24 hours costs very little. An RPO of five minutes requires continuous replication. An RPO of zero requires synchronous replication with a latency and complexity penalty on every write. Present the options with their costs and let the business choose.
Set them per system, not for everything
A single organisation-wide RTO is almost always wrong. The customer-facing application, the billing system, the internal analytics warehouse and the marketing site have very different tolerances.
Classify systems into three tiers. Tier one is revenue-affecting and customer-visible, with an aggressive RTO. Tier two matters within a day. Tier three can wait a week. Most estates are 10% tier one and 60% tier three, and recognising that saves a great deal of money.
The four recovery strategies
Ordered by cost and speed.
Backup and restore. Data backed up, infrastructure rebuilt from code when needed. Cheapest; RTO measured in hours to days. Appropriate for tier three and often for tier two.
Pilot light. Core data replicated continuously to a second region, minimal infrastructure running, the rest provisioned on failover. RTO in tens of minutes at modest cost.
Warm standby. A scaled-down but complete copy running in the second region, scaled up on failover. RTO in minutes; cost is meaningful but far below a full duplicate.
Active-active. Both regions serving traffic. Near-zero RTO and RPO, at roughly double the infrastructure cost plus significant complexity in data consistency.
Most SaaS products are well served by pilot light for tier one and backup-and-restore for everything else. Active-active is justified by contractual uptime commitments, not by ambition.
Backups: the part that is usually wrong
Four properties separate backups that work from backups that exist.
- Automated. Anything requiring a human to remember will eventually not happen.
- Isolated. In a separate account or project with separate credentials. Backups an attacker can delete with the same access they used to encrypt your production data are not backups. Ransomware specifically targets backup systems.
- Immutable where possible. Object lock or equivalent, so even an administrator cannot delete within the retention window.
- Tested by restoring. Not by checking that the backup job reported success. The number of organisations that discover an unreadable backup during an incident is not small.
Retention should be defined by what you might need: daily for a month, weekly for a quarter, monthly for a year is a common pattern, adjusted for any contractual or regulatory requirement.
Writing the runbook
The plan document is not the runbook. The plan explains strategy and objectives; the runbook is the thing someone follows at 3am.
A usable runbook has: numbered steps in order, the exact commands or console paths, expected output at each step so the reader knows it worked, decision points with clear criteria, and a named role for each step rather than a person’s name.
Write it for someone competent but unfamiliar. Avoid “restore the database” in favour of the actual procedure. Store it somewhere that does not depend on the systems being recovered — a plan living only in a wiki hosted on the affected infrastructure has failed before it starts.
What the plan must also cover
Technical recovery is roughly half of it. The rest:
- Who declares a disaster and on what criteria. Ambiguity here costs the first hour.
- Communication. Who tells customers, through which channel, how often. A status page hosted on your own infrastructure is not a status page.
- Escalation and contacts. Including cloud provider support, with the account details needed to raise a priority ticket.
- Failback. How to return to normal once the primary is healthy. This is frequently omitted and is often harder than the failover.
- Data reconciliation. What to do about writes that happened during the gap.
Testing without breaking production
Three levels, escalating.
Tabletop. The team walks through a scenario verbally. Cheap, surfaces missing steps and unclear ownership, and finds a surprising amount. Do this quarterly.
Component restore. Restore a database backup to an isolated environment and verify the data. Monthly, automated if possible. This is the test that catches corrupt backups.
Full failover exercise. Actually fail over to the secondary, with traffic, in a planned window. Annually at minimum. This is the only test that proves the plan, and it is the one organisations avoid for years.
Record what went wrong in each exercise and fix it. A test that finds no problems usually means the test was not realistic.
Keeping it current
Plans decay faster than almost any other document, because infrastructure changes weekly and the plan is reviewed annually. Two habits help: tie a plan review to any significant architecture change, and put the runbook in version control next to the code so it shows up in reviews rather than sitting in a document nobody opens.
Auditors will ask for evidence of testing as part of SOC 2 availability criteria, which is a useful external forcing function if internal discipline is hard to maintain.
The scenarios worth planning for
Disaster recovery plans often assume a regional outage, which is the least likely scenario. Four others are more common and need different responses.
Accidental deletion. Someone drops a table or deletes a bucket. Replication does not help; point-in-time backups do. This is the most frequent real-world data loss event.
Data corruption. A bad deployment writes incorrect data for six hours before anyone notices. Recovery requires knowing when it started and having a backup from before it, which is an argument for longer retention than the minimum.
Account compromise. An attacker with administrative credentials can delete infrastructure and backups in the same account. This is the scenario that justifies isolated backup accounts and immutable retention.
Dependency failure. A third-party service you rely on goes down and takes your product with it. No amount of multi-region helps if your authentication provider is unreachable. Identify single points of failure outside your infrastructure and decide, explicitly, whether to accept the risk or build a fallback.
Communicating during an incident
Customers judge an outage largely by how it was communicated. Three rules hold up under pressure.
Acknowledge early, before you understand the cause. “We are aware of an issue affecting logins and are investigating” within ten minutes buys far more goodwill than a detailed explanation two hours later.
Update on a stated cadence, even when there is nothing new. Silence reads as loss of control.
Avoid estimates you cannot keep. A missed restoration estimate does more damage than no estimate. Say what you know, what you are doing, and when you will next update.
Frequently asked questions
What is a reasonable RTO for a SaaS product?
Four hours is a common target for the main application in mid-market software, with tighter objectives where contracts specify uptime. The right answer depends on what customers do with the product and what your agreements promise.
Does multi-region hosting replace disaster recovery?
No. Multi-region protects against infrastructure failure. It does not protect against data corruption, accidental deletion or a compromised account, all of which replicate faithfully to every region.
How often should backups be tested?
Restore at least monthly, automated where possible. A full failover exercise at least annually.
Who should own the plan?
One named engineering owner for the technical content and one business owner for the objectives and communication. Shared ownership with no name attached produces a document that is never updated.
