Backup & Disaster Recovery
A backup you've never restored isn't a backup — it's an unverified assumption about what your data looked like at some point in the past.
Overview
Every storage decision covered in this journey — replication, erasure coding, tiering, consensus — assumes the system stays up. Backup and disaster recovery is the discipline for the case where all of that fails anyway: hardware dies, an operator runs a bad migration, or an entire region goes down, and the only thing left is whatever was backed up.
Why It Exists
Replication (covered earlier) protects against a single node failing, but it doesn't protect against a bad write, a bug, or a `DROP TABLE` propagating instantly to every replica — replication keeps copies in sync, which is exactly the problem when the thing being synced is itself wrong. Backup and disaster recovery exists as a genuinely independent layer of protection, specifically because it captures point-in-time snapshots of the past, deliberately decoupled from whatever is happening to the live system right now.
Real World Example
An engineer runs a migration script against production that accidentally deletes a table's data. Every read replica faithfully replicates that deletion within seconds — replication did exactly what it's designed to do, and it just propagated the disaster. The only way to recover is a point-in-time recovery: restoring the database to the exact moment before the migration ran, using a base backup plus the write-ahead log records leading up to (but not past) that moment.
Example Data
RPO and RTO for the same incident, under two different backup strategies
| Backup Strategy | RPO (data loss window) | RTO (downtime) |
|---|---|---|
| Nightly full backup only | Up to 24 hours of data lost | Hours (restoring a full backup) |
| Base backup + continuous WAL archiving | Seconds to minutes of data lost | Minutes to an hour (replay WAL to the target moment) |
The Vocabulary and Mechanics of Recovery
RPO — Recovery Point Objective
How much data loss is acceptable, measured as a window of time — an RPO of 5 minutes means the business accepts losing at most 5 minutes of writes in a disaster, which directly determines how frequently backups (or WAL archiving) must run.
RTO — Recovery Time Objective
How much downtime is acceptable while actually recovering — an RTO of 1 hour means the restore process, end to end, must be able to bring the system back within that hour, which shapes decisions like how large backups are and how practiced the restore procedure is.
Point-in-Time Recovery (PITR)
Restoring a database to an arbitrary specific moment (not just the last full backup) by taking a base backup and replaying write-ahead log records forward up to the target timestamp — this is precisely what makes recovering to 'right before the bad migration' possible.
The 3-2-1 Backup Rule
A widely-used baseline: keep at least 3 copies of data, on at least 2 different storage media, with at least 1 copy stored off-site — designed so that no single failure (a disk, a data center, a region) can take out every copy at once.
Diagram
Point-in-time recovery: a base backup plus WAL replay to an exact target moment
Base backup taken
e.g. midnight
WAL records archived continuously
every committed write since midnight
Bad migration runs at 14:32
target recovery point: 14:31:59
Restore: base backup + replay WAL up to 14:31:59
stop before the bad write
Common Mistakes
Relying on replication alone as a backup strategy
Why: Replication propagates every write, including bad ones, to every replica almost immediately — it protects against hardware failure, not against data corruption caused by human error or a bug.
Fix: Maintain independent, point-in-time-capable backups in addition to replication, since they solve different failure modes.
Taking backups regularly but never actually testing a restore
Why: A backup can be silently corrupted, incomplete, or missing a critical piece (like WAL archives needed for PITR) for a long time before anyone notices — usually right when it's needed most.
Fix: Periodically perform full test restores from real backups, on a schedule, as a first-class operational practice, not an afterthought.
Setting an RPO and RTO without designing the actual backup strategy to meet them
Why: An RPO of '5 minutes of acceptable data loss' is meaningless if backups only run nightly — the stated objective and the actual technical capability need to match, or the numbers are just aspirational.
Fix: Choose the backup frequency, WAL archiving, and restore procedure specifically to satisfy the RPO/RTO the business has actually committed to, and verify it under real restore drills.
Interview Questions
What's the difference between RPO and RTO?
RPO (Recovery Point Objective) is how much data loss is acceptable, measured as a time window — how far back you might have to roll. RTO (Recovery Time Objective) is how much downtime is acceptable while actually performing the recovery — how long the restore process is allowed to take.
Why isn't database replication a substitute for backups?
Replication is designed to keep replicas in sync with the primary as fast as possible — which means a bad write, a bug, or an accidental deletion gets replicated too, almost instantly. Backups are deliberately decoupled point-in-time snapshots of the past, which is exactly what's needed to recover from a disaster that replication itself just faithfully propagated.
Your company sets an RPO of 1 minute and an RTO of 15 minutes for a critical database. What does that actually require of your backup architecture, and how would you validate it's actually being met?
An RPO of 1 minute requires continuous WAL archiving (not just periodic full backups) so that point-in-time recovery can restore to within a minute of any failure — nightly or even hourly full backups alone wouldn't come close. An RTO of 15 minutes requires the restore procedure itself — provisioning a new instance, restoring the base backup, and replaying WAL — to complete end-to-end within that window, which likely means keeping base backups small/recent enough that WAL replay doesn't take too long, and having the restore procedure automated and pre-tested rather than manual. To validate it, I'd run regular disaster-recovery drills that simulate an actual failure and time the real restore against both objectives, since an RPO/RTO that's never been tested under realistic conditions is just an assumption, not a guarantee.
Production Best Practices
Do
✓Maintain independent, point-in-time-capable backups in addition to replication.
✓Periodically perform full test restores from real backups on a defined schedule.
✓Design backup frequency and restore procedure specifically to meet a stated RPO/RTO, and validate it with drills.
Don't
✗Don't rely on replication alone as a substitute for backups.
✗Don't take backups without ever testing whether they can actually be restored.
✗Don't set an RPO/RTO without verifying the backup architecture can actually meet it.
Comparison
| Protects Against | Data Freshness | Recovers From Bad Writes? | |
|---|---|---|---|
| Replication | Node/hardware failure | Near real-time | No — propagates them too |
| Backups + PITR | Data corruption, human error, region loss | Point-in-time, as of last backup/WAL | Yes — restore to before the bad write |