PostgreSQL Disaster Recovery on AWS

Part I · Disaster Recovery Foundations

01RPO & RTO — the two dials of disaster recovery★ start here▶

Scenario: the database is gone. Your boss asks two questions: "how much data did we lose?" and "how long until we're back?" Every DR decision — backup frequency, replica placement, budget — is really an answer to those two numbers. Name them first, then build to hit them.

What

RPO (Recovery Point Objective) = how much data you can lose, in time. RTO (Recovery Time Objective) = how long you can be down. Two targets, measured in minutes/hours.

Why

They turn "don't lose data" into a design and a budget. Tighter RPO/RTO cost more; the business decides the number, engineering builds to it.

How

RPO is set by backup/replication frequency; RTO by how fast you can restore or promote. Pick per system — a ledger ≠ an analytics cache.

The two windows around a disaster

              last good copy            DISASTER            back online
   ───────────────●───────────────────────╳───────────────────●──────────►
                   \_______ RPO _______/  \_______ RTO _______/
                    data written here       downtime here
                    is LOST on recovery      until service returns

   nightly backup → RPO up to 24h        restore 2TB from S3 → RTO hours
   WAL every 1 min → RPO ~1 min          promote hot standby → RTO seconds
   sync replica    → RPO ~0              (choose per system & per budget)

🧠

Analogy 1 — saving a document you're writing: RPO is "how much work since my last save can I bear to redo?" — save every hour and a crash costs an hour. RTO is "how long to reopen the app and get back to where I was?" DR is choosing both, on purpose, before the crash.

🚑

Analogy 2 — an ambulance response: RTO is how fast help arrives (downtime); RPO is how much was lost before it did. You pay more for a faster ambulance and a closer hospital — exactly the RTO/RPO cost curve.

RPO/RTO drive the architecture — and the bill

# RPO — how much DATA you can afford to lose (measured in time)
#   nightly logical dump only     → up to 24h of writes gone
#   WAL archived every 1 min      → ~1 min of writes at risk
#   synchronous standby           → ~0 (no acknowledged write is lost)
#
# RTO — how long you can be DOWN before recovery
#   restore a big backup from S3  → minutes to hours (size-dependent)
#   promote a warm/hot standby    → seconds to a few minutes
#
# write the two numbers per system FIRST; they pick every other decision.

🧒

In plain wordsTwo questions decide your whole backup plan. RPO: if things blow up, how much recent work can you afford to lose — a day? a minute? none? RTO: how long can the app be down before it's a real problem — hours? seconds? Smaller answers cost more money and effort, so the business picks the numbers and engineers build to reach them.

✅ Do

Write an explicit RPO and RTO for each system before designing DR
Let the business own the numbers — they imply cost and complexity
Re-derive backup frequency and replica strategy from the targets

❌ Don't

Say "we can't lose any data" for everything — near-zero RPO is expensive
Confuse RPO (data loss) with RTO (downtime) — they need different controls

Gotcha: an untested RTO is a fantasy. Teams quote "RTO: 1 hour" from a slide, then discover during a real outage that restoring 2TB from S3, replaying WAL, warming caches, and repointing apps takes six. Your true RTO is whatever your last timed restore drill (topic 11) measured — everything else is a hopeful guess.

02What you're recovering from — the failure taxonomy▶

Scenario: you built a beautiful multi-AZ setup and feel safe — then an engineer runs DELETE FROM orders without a WHERE, and your standby faithfully replicates the deletion in milliseconds. High availability didn't help, because you were defending the wrong failure. Different disasters need different defenses.

What

The categories of disaster: hardware/AZ failure, region outage, data corruption, human error (bad DELETE/migration), ransomware/malice, and accidental deletion of the DB itself.

Why

Each needs a different control. Replication survives a dead server but happily copies a bad DELETE. Only point-in-time backups undo human error.

How

Map each failure to a defense: HA for hardware, cross-region for region loss, PITR/immutable backups for human error and ransomware.

Different failures, different defenses

  FAILURE                        WHAT SAVES YOU
  ─────────────────────────      ───────────────────────────────
  server/disk dies               Multi-AZ standby (HA)        t13
  whole AWS region down          cross-region replica/backup  t14,17,18
  bad DELETE / migration         Point-In-Time Recovery       t8,15
  data corruption spreads        PITR to BEFORE the corruption
  ransomware / malicious wipe    immutable, cross-account copies  t18
  someone drops the database     backups that live ELSEWHERE
  → replication does NOT fix the bottom four — it copies them faithfully

🧠

Analogy 1 — smoke alarm vs flood insurance vs a spare key: each protects a different disaster. A smoke alarm won't help in a flood; a spare key won't help against fire. "Being safe" isn't one thing — it's covering each distinct way things go wrong.

🪞

Analogy 2 — a mirror copies your mistakes: a replica is a live mirror. Cut yourself shaving and the mirror shows it instantly — it never prevents the cut. To undo a mistake you need a photo from before it (a backup in time), not a mirror.

The failure that replication makes worse, not better

# human error, replicated in milliseconds to every standby:
$ psql -c "DELETE FROM orders;"      # forgot the WHERE clause
DELETE 4821993                        # ...and the hot standby now agrees

# HA (multi-AZ) gives you a perfect copy of the WRONG state.
# only a copy from BEFORE the command saves you:
$ # restore to 10:41:59, one second before the delete  (PITR, topic 8)
$ #   recovery_target_time = '2026-07-14 10:41:59+00'

🧒

In plain words"Disaster" isn't one thing. A server dying, a whole data center going dark, someone accidentally deleting all the data, or a hacker wiping it — these are different problems needing different backups. The big trap: a live copy (replica) instantly copies mistakes too, so it protects you from a broken machine but not from a bad command. You need time-travel backups for that.

✅ Do

List the failure modes and map each to a specific control
Keep point-in-time backups to undo human error and corruption
Store at least one copy immutably and in another account/region

❌ Don't

Assume replication or Multi-AZ is "backup" — it copies mistakes instantly
Defend only the dramatic failures (region loss) and skip the common one (a bad DELETE)

Gotcha: the most likely disaster isn't a meteor hitting us-east-1 — it's a human running a bad migration or an app deleting rows it shouldn't. These "logical" failures replicate instantly to every standby and every synchronous replica, so the fancier your HA, the faster the mistake spreads. The only defense is a backup that captures a point before the error — which is why PITR (topic 8) is the backbone of real DR.

03Backup vs replication vs DR — three different jobs★ core idea▶

Scenario: "We have a replica, so we're backed up." No. A replica gives you a hot copy for failover, but if a row is corrupted or deleted, the replica has the corruption too. Backup, replication, and DR are three distinct capabilities — and you need all three, because each fails where the others cover.

What

Backup = a point-in-time copy you can restore from the past. Replication = a live, up-to-date copy for availability. DR = the whole plan (people, steps, targets) to recover from disaster.

Why

Confusing them is the classic fatal mistake. Replication protects against a dead server; backups protect against a bad write; DR is what ties strategy, testing, and runbooks together.

How

Run all three: streaming replicas for HA/RTO, PITR backups for RPO and human-error recovery, and a written, tested DR plan over both.

Three jobs, no substitutes

  BACKUP            REPLICATION          DR (the plan)
  a copy from       a live copy for      strategy + runbook +
  the PAST          NOW / failover       testing over both
  ───────────       ───────────────      ─────────────────────
  undo a mistake    survive a crash      hit RPO & RTO on purpose
  restore in time   promote in seconds   people, comms, drills
  protects RPO      protects RTO         protects the BUSINESS

  a replica is NOT a backup (it copies corruption)
  a backup is NOT HA (restoring takes time)
  neither is DR without a tested plan wrapping them

🧠

Analogy 1 — a photocopy, a live video feed, and a fire drill: a backup is a photocopy filed away (go back to it). A replica is a live camera feed (see exactly what's happening now, switch to it). DR is the fire drill — the practiced plan that uses both when the alarm rings. You need all three.

👯

Analogy 2 — a twin vs a diary: a replica is your identical twin doing everything you do in real time — great if you're hit by a bus, useless for "what did I have yesterday?" A backup is a diary you can page back through. Different questions, different tools.

You can tell them apart by what each can and can't undo

# REPLICATION: check the live standby is caught up (availability)
$ psql -c "SELECT client_addr, state, replay_lag FROM pg_stat_replication;"
#   → survives a primary crash; does NOT let you go back in time

# BACKUP: restore yesterday's state (recover from a mistake)
$ pg_restore -d shop_restored /backups/shop_2026-07-13.dump
#   → undoes a bad DELETE; does NOT give instant failover

# DR: the plan that says WHICH to use, WHO runs it, and the target
$ cat dr-runbook.md    # RPO/RTO, promote steps, comms, contacts (topic 24)

🧒

In plain wordsThree different safety nets, often confused: a backup is a saved copy from the past you can go back to (undo mistakes). Replication is a live twin ready to take over instantly if a machine dies (but it copies mistakes too). DR is the written, practiced plan that uses both when things go wrong. "We have a replica" is not "we're backed up" — you need each.

✅ Do

Run replication for availability and backups for recoverability
Wrap both in a written, tested DR plan with owners and targets
Say precisely which capability protects which failure

❌ Don't

Call a replica a backup — it faithfully replicates corruption and deletes
Assume having backups means you have DR — DR is the tested plan around them

Gotcha: "the replica is our backup" has ended companies. Replication is synchronous with your mistakes — a DROP TABLE, a corrupting bug, or ransomware reaches the standby as fast as the network allows. Backups are the only copy that exists at a different point in time, which is exactly what you need to recover from anything logical. Keep them separate, and keep at least one backup somewhere the primary's credentials can't reach.

04The DR strategy spectrum — from restore to active-active▶

Scenario: two systems, two very different needs. Your internal wiki can be down for hours and lose a day — a nightly backup is plenty. Your payments ledger can lose nothing and must fail over in seconds. Same company, opposite DR strategies. AWS names four points on this spectrum; you pick per system.

What

Four DR patterns, cheapest/slowest to priciest/fastest: Backup & Restore → Pilot Light → Warm Standby → Multi-Site Active-Active.

Why

Cost and complexity climb as RTO/RPO shrink. Matching the pattern to each system's targets avoids both under-protecting the ledger and overspending on the wiki.

How

Pick per workload from its RPO/RTO: hours-and-a-day → restore; minutes → warm standby; near-zero → active-active (and a big bill).

Cheaper/slower ← → pricier/faster

  BACKUP & RESTORE    PILOT LIGHT        WARM STANDBY       ACTIVE-ACTIVE
  ─────────────────   ──────────────     ───────────────    ──────────────
  backups in S3;      data replicated;   scaled-down full   full stacks in
  spin up on disaster core off till need running copy       2+ regions, live
  RTO: hours          RTO: 10s of min    RTO: minutes       RTO: ~0
  RPO: hours          RPO: minutes       RPO: seconds       RPO: ~0
  $                   $$                 $$$                $$$$
  wiki / analytics    typical apps       important apps     the ledger

🧠

Analogy 1 — spare-tire options: a can of sealant in the trunk (backup/restore — cheap, slow, gets you limping). A skinny donut spare (pilot light). A full-size spare mounted and inflated (warm standby). A second car idling in the driveway (active-active). You buy the level of readiness the trip demands.

🏥

Analogy 2 — backup power: flashlights in a drawer (restore), a generator you must go start (pilot light), a generator already running on low (warm standby), or two independent grids (active-active). Hospitals pay for the last; a shed uses flashlights.

Choose the pattern from the numbers, per system

# internal wiki:  RTO hours, RPO 1 day  → BACKUP & RESTORE
#   nightly RDS snapshot; on disaster, restore into the DR region

# customer app:   RTO minutes, RPO seconds → WARM STANDBY
#   cross-region read replica kept warm; promote + repoint on disaster

# payments ledger: RTO ~0, RPO ~0 → ACTIVE-ACTIVE
#   Aurora Global Database (topic 17) or multi-region writers; expensive
#
# one company runs ALL of these at once — matched to each system's targets.

🧒

In plain wordsThere's a menu of DR plans from cheap-and-slow to expensive-and-instant: just keep backups and rebuild when needed; keep a tiny copy warm; keep a smaller full copy running; or run two live copies in different regions at once. You don't pick one for the whole company — you match each app to how fast it must recover and how much it can lose.

✅ Do

Pick a pattern per system from its RPO/RTO, not one size for all
Start cheap; move up the spectrum only where the targets demand it
Cost the higher tiers honestly — active-active roughly doubles infra

❌ Don't

Build active-active for a system that can tolerate a nightly restore
Put the revenue-critical ledger on backup-and-restore to save money

Gotcha: the tiers describe infrastructure readiness, not data recoverability. Even full active-active does not protect you from a logical error — a bad write replicates to every live site instantly. So every tier still needs point-in-time backups underneath it. The spectrum buys you RTO (how fast you're back); PITR buys you the ability to go back to a good moment at all.

05Durability & the 3-2-1 backup rule▶

Scenario: your backups run nightly to a folder on the same server as the database. The server's disk fails — and takes the database and every backup with it. Or the AWS account is compromised and the attacker deletes the DB and its snapshots in the same breath. A backup that shares fate with the thing it protects isn't a backup.

What

The 3-2-1 rule: keep 3 copies of data, on 2 different media/systems, with 1 off-site (different region/account) — ideally one immutable.

Why

Backups fail together when they share fate — same disk, same account, same region, same credentials. Independence is what makes a copy survive the disaster that killed the original.

How

Automated backups + a copy in another region + a copy in another account (or object-lock/immutable), so no single failure or actor can take them all.

3-2-1 — copies that don't share fate

   ● primary DB (region A, account A)
   ● backup #2  (region A, account A)   ← same account: attacker deletes both
   ● backup #3  (region B, account B, object-lock/immutable)  ← survives

   3 copies · 2 systems (RDS + S3) · 1 off-site & immutable
   test: "could ONE failure or ONE compromised credential take them all?"
         if yes → they share fate → it's not 3-2-1 yet

🧠

Analogy 1 — not keeping your only spare key in the car: a spare key locked inside the same car helps no one. You keep one on you, one at home, one with a neighbor. Backups are spare keys: value comes from them being in independent places.

🥚

Analogy 2 — eggs in different baskets, in different rooms: two baskets on the same shelf both fall when the shelf breaks. Real safety is baskets in different rooms owned by different people — so one accident, fire, or thief can't reach them all.

Independence, made concrete on AWS

# copy #2: automated backups in the primary region/account
$ aws rds describe-db-snapshots --db-instance-identifier shop-prod

# copy #3: replicate to a DIFFERENT region AND account, immutably
$ aws rds copy-db-snapshot \
    --source-db-snapshot-identifier shop-prod-2026-07-14 \
    --target-db-snapshot-identifier shop-dr-2026-07-14 \
    --kms-key-id alias/dr-key --region us-west-2        # cross-region
# + S3 backups with Object Lock (WORM) in a locked-down backup account
$ aws s3api put-object-lock-configuration --bucket pg-dr-backups ...  # immutable

🧒

In plain wordsDon't keep your only backup next to the thing it's backing up — if a fire (or a hacker, or a bad disk) hits, it takes both. The rule of thumb: keep three copies, on two kinds of storage, with one far away (different region and account) and locked so it can't be deleted. Then no single accident or attacker can wipe out everything at once.

✅ Do

Keep at least one backup in a different region and account
Make one copy immutable (S3 Object Lock / vault lock) against ransomware
Ask "could one failure or one credential delete them all?" — fix it if yes

❌ Don't

Store backups on the same host/disk/account as the database
Give the app's or admin's normal credentials delete rights on backups

Gotcha: ransomware and malicious insiders specifically hunt your backups first — an attacker with account admin deletes the database and its snapshots together, then demands payment. Same-account, same-credential backups are exactly what they count on. The immutable, cross-account copy (S3 Object Lock, a separate "backup" account with tightly scoped access) is the one that survives, and it's the difference between "restore and move on" and "pay the ransom."

Part II · PostgreSQL Backup & Recovery Mechanics

06Logical backups — pg_dump & pg_restore▶

Scenario: you need to copy one database to a staging environment, or restore a single accidentally-dropped table — not the whole cluster. A logical backup exports the data as SQL/objects you can restore selectively, into any compatible Postgres. It's portable and surgical, but it's slow to make and restore at scale.

What

pg_dump exports one database as a logical set of SQL statements or a compressed archive; pg_dumpall adds cluster-wide objects (roles, tablespaces). pg_restore loads an archive back.

Why

It's portable (across versions/platforms) and selective (one table, one schema) — ideal for migrations, dev refreshes, and single-object recovery.

How

Use the custom/directory format (-Fc/-Fd) for compression and parallel restore; it's a consistent snapshot at dump-start time.

Logical dump = a portable recipe to rebuild the data

   running DB ──pg_dump -Fc──► shop.dump  (compressed archive)
                                  │  contains: schema + data as objects
                                  ▼
   fresh DB ◄──pg_restore -j 4───┘  (parallel, selective: -t one_table)

   ✓ portable across versions/OS      ✗ slow to dump/restore huge DBs
   ✓ restore ONE table                ✗ RPO limited to dump frequency
   ✓ human-diffable (plain format)    ✗ not point-in-time (topic 8 is)

🧠

Analogy 1 — a recipe vs a photograph: a logical dump is the recipe — the instructions to recreate the dish in any kitchen. It travels anywhere and you can copy just one dish's steps, but cooking it all again takes time. (A physical backup, topic 7, is a photograph of the finished plate.)

📦

Analogy 2 — flat-pack furniture: pg_dump ships the data as labeled parts and instructions — compact and portable, assembles into any compatible house. But assembling a mansion from flat-pack (restoring a huge DB) is slow compared to lifting a pre-built room into place.

Dump in the custom format; restore in parallel or selectively

# ✅ custom format: compressed, restore in parallel, restore ONE object
$ pg_dump -Fc -d shop -f shop.dump              # consistent as of dump start
$ pg_restore -d shop_new -j 4 shop.dump          # parallel restore

# restore just one accidentally-dropped table (surgical recovery)
$ pg_restore -d shop -t orders --data-only shop.dump

# cluster-wide roles/grants live outside pg_dump — capture them too
$ pg_dumpall --globals-only > globals.sql        # roles, tablespaces
# ❌ pg_dump shop > shop.sql                        # plain SQL: no parallelism

🧒

In plain wordsA logical backup is like writing down the full recipe for your database — every table and row as instructions to rebuild it. It's great because you can carry it to any computer and even restore just one dish (one table). The downside: for a giant database, cooking the whole thing from the recipe is slow, and it only captures the moment you started writing it down.

✅ Do

Use -Fc (custom) or -Fd (directory) for compression and parallel restore
Capture globals separately with pg_dumpall --globals-only
Use logical dumps for migrations, dev refreshes, and single-object recovery

❌ Don't

Rely on nightly pg_dump alone for a large, low-RPO production DB
Forget roles/grants — a data-only restore into an empty cluster fails without them

Gotcha: a pg_dump is consistent as of the moment it started, and on a large busy database it can run for hours holding that snapshot — so your real RPO is "last night's dump," and restore time (RTO) grows with data size. For anything with a tight RPO/RTO, logical dumps are a supplement (portability, selective restore), not the backbone. The backbone is physical backups + WAL (topics 7–9).

07Physical backups & the WAL▶

Scenario: your database is 2TB and needs a tight RPO. A nightly logical dump can't keep up. You need a physical backup — a byte-level copy of the data files — plus the Write-Ahead Log, the stream of every change, which together let you restore fast and roll forward to any second. This pairing is the heart of serious Postgres DR.

What

A physical (base) backup is a copy of the actual data directory (via pg_basebackup). The WAL (Write-Ahead Log) records every change before it's applied — the redo stream.

Why

Base backup + archived WAL = fast restore and the ability to replay forward to any point (PITR, topic 8). It's how replication and low-RPO recovery both work.

How

Take a base backup, then continuously archive WAL segments. Restore = lay down the base, then replay WAL up to your target time.

Base backup + WAL = a restorable point in time

   BASE BACKUP (byte copy of data dir)      WAL stream (every change, in order)
   ┌───────────────────────────┐           w1 w2 w3 w4 w5 w6 w7 w8 ...
   │  full snapshot @ 02:00      │  ─────►  ───────────────────────────►
   └───────────────────────────┘           each segment archived off-box

   restore = copy base @02:00, then REPLAY WAL forward to your target
   (WAL is also what a streaming replica consumes live, topic 10)
   lose the WAL and you're stuck at the base backup's moment.

🧠

Analogy 1 — a photo plus the security-camera footage since: the base backup is a photo of the room at 2am. The WAL is the continuous camera feed of every change after. Together you can reconstruct the room at any second — restore the photo, then fast-forward the footage to the moment you want.

🧾

Analogy 2 — an account statement plus every receipt since: the base backup is last month's statement; the WAL is every receipt after it, in order. Replaying the receipts onto the statement reproduces your exact balance at any chosen moment.

A base backup, and the WAL that rolls it forward

# physical base backup — byte-level copy of the data directory
$ pg_basebackup -h primary -D /backup/base -Fp -Xs -P -R
#   -Xs stream WAL alongside · -R writes standby config · -P progress

# WAL is written continuously; archive each segment somewhere durable:
#   postgresql.conf
#     wal_level = replica
#     archive_mode = on
#     archive_command = 'aws s3 cp %p s3://pg-wal/%f'   # ship each segment
#
# restore later = restore base, then set a restore_command to fetch WAL
#   restore_command = 'aws s3 cp s3://pg-wal/%f %p'      # replay forward

🧒

In plain wordsFor a big database, instead of re-writing the whole recipe every night, you take one full snapshot of the actual files, and then keep a running logbook of every single change (the WAL) as it happens. To recover, you put back the snapshot and then replay the logbook forward to whatever second you want. That logbook is also how live copies (replicas) stay in sync.

✅ Do

Base backup + continuous WAL archiving for large/low-RPO databases
Archive WAL to durable, off-box storage (S3) — the WAL is half your recovery
Prefer a tool (pgBackRest/WAL-G, topic 9) over hand-rolled archive scripts

❌ Don't

Keep WAL only on the primary's disk — lose the disk, lose the roll-forward
Assume a base backup alone is enough — without WAL you're frozen at its timestamp

Gotcha: the WAL is the fragile, precious half. If archive_command fails silently (S3 perms, full disk) and nobody's watching, WAL segments pile up on the primary — filling its disk and, worse, leaving a gap so you can only recover to the last successfully-archived segment. Monitor archiving success and pg_wal disk usage as first-class DR health signals; a broken archiver quietly destroys your RPO.

08Point-in-time recovery (PITR)★ core idea▶

Scenario: at 10:42 someone ran a bad migration that corrupted the orders table. You don't want last night's backup (you'd lose a day) and you can't use the replica (it copied the corruption). You want the database exactly as it was at 10:41:59 — one second before. That's point-in-time recovery, and it's the single most important DR capability.

What

PITR = restore a base backup, then replay archived WAL forward and stop at a chosen time (or transaction), landing the database at that exact moment.

Why

It's the only way to undo logical disasters — bad writes, migrations, corruption — that replication faithfully copies. It converts "we lost a day" into "we lost one second."

How

Set recovery_target_time (or LSN/xid), point restore_command at your WAL archive, and start recovery; Postgres replays up to the target and stops.

Replay the WAL — but stop one second before the mistake

   base @02:00 ──replay WAL──► ... 10:41:58  10:41:59 │ 10:42:00 BAD MIGRATION
                                                       ▲
                          recovery_target_time = '10:41:59'  ← STOP here

   result: a database identical to the instant BEFORE the corruption.
   (choose target by time, by transaction id, or by a named restore point)

🧠

Analogy 1 — a movie you can pause one frame early: the base backup starts the film; the WAL plays it forward frame by frame. PITR is pausing exactly on the frame before the disaster — you get the whole story up to that instant and nothing after.

↩️

Analogy 2 — undo, but for the entire database: a word processor's undo walks back keystroke by keystroke. PITR is undo for the whole cluster — it rebuilds every table to a chosen moment, so a catastrophic command becomes a recoverable "back up ten minutes."

Restore to the exact second before the damage

# 1) restore the most recent base backup taken BEFORE the target time
$ cp -a /backup/base/* $PGDATA/

# 2) tell Postgres where to fetch WAL and WHEN to stop
#    postgresql.conf / recovery settings:
#      restore_command = 'aws s3 cp s3://pg-wal/%f %p'
#      recovery_target_time = '2026-07-14 10:41:59+00'
#      recovery_target_action = 'promote'
$ touch $PGDATA/recovery.signal      # PG 12+: signals recovery mode
$ pg_ctl start                        # replays WAL up to 10:41:59, then promotes

# verify you landed before the damage, THEN repoint the app
$ psql -c "SELECT count(*) FROM orders;"   # sanity-check the row count

🧒

In plain wordsPITR is a time machine for your database. If a mistake happened at 10:42, you tell Postgres "rebuild yourself exactly as you were at 10:41:59" — it lays down a snapshot and replays the change-logbook forward, stopping one second before the disaster. Instead of losing a whole day (last backup) or copying the mistake (replica), you lose almost nothing. This is the most important recovery trick to know.

✅ Do

Keep base backups + continuous WAL so any recent second is reachable
Restore to just before the bad event; verify before repointing the app
Know your target options: time, transaction id (xid), or named restore point

❌ Don't

Recover onto the damaged instance — restore to a NEW one, then cut over
Guess the target time — use logs/audit to find the exact moment of harm

Gotcha: PITR only reaches as far back as your oldest retained base backup and as far forward as your last archived WAL — a gap in either end silently shrinks your recovery window. And the target must sit after a base backup and within continuous WAL: if you keep 7 days of backups but 3 days of WAL, you can't PITR to day 5. Retention of base backups and WAL must be planned together, or your time machine has holes.

09Continuous WAL archiving — pgBackRest & WAL-G▶

Scenario: you tried hand-rolling archive_command = 'cp %p /mnt/wal/%f' and a dozen things went wrong — a full disk left a gap, no compression blew your storage budget, and you had no easy way to verify or expire old backups. Purpose-built backup tools solve exactly these, which is why nobody runs serious Postgres DR on shell scripts.

What

Backup managers (pgBackRest, WAL-G) that automate base backups + WAL archiving to object storage, with compression, encryption, parallelism, verification, and retention.

Why

They turn fragile hand-written archiving into a reliable system — atomic backups, integrity checks, incremental/differential backups, and one-command restore/PITR.

How

Configure a repository (S3), set archive_command to the tool, schedule full/incremental backups, and let it manage expiry to your retention policy.

A managed pipeline instead of a brittle script

   primary ──archive_command = pgbackrest──► S3 repo (compressed, encrypted)
      │  full / differential / incremental backups on a schedule
      │  ├─ integrity verified (checksums)
      │  ├─ retention: keep 2 full + WAL to satisfy PITR window
      └─ restore: `pgbackrest restore --type=time --target=...`  (one command)

   vs.  archive_command = 'cp %p /mnt/wal/%f'
        → no compression, no verify, no retention, silent gaps on full disk

🧠

Analogy 1 — a professional moving company vs carrying boxes yourself: you could move a house one armful at a time (a shell script), but a pro crew packs, labels, insures, and inventories everything — and can put it back correctly. pgBackRest is the crew; cp is your arms.

🏦

Analogy 2 — a bank's vault vs cash under the mattress: both "store money," but the vault adds logging, verification, access control, and recovery guarantees. A managed backup tool is the vault for your WAL; a copy script is the mattress.

Point Postgres at the tool; restore is one command

# postgresql.conf — let the tool own archiving
#   archive_command = 'pgbackrest --stanza=shop archive-push %p'

# take backups on a schedule (full weekly, incremental daily)
$ pgbackrest --stanza=shop --type=full backup
$ pgbackrest --stanza=shop --type=incr backup
$ pgbackrest --stanza=shop check          # verify archiving + backups are healthy

# point-in-time restore — the tool fetches the right base + WAL for you
$ pgbackrest --stanza=shop --type=time \
    --target="2026-07-14 10:41:59+00" restore
# WAL-G is a similar tool (Go, cloud-native): wal-g backup-push / backup-fetch

🧒

In plain wordsCopying your database's change-logs by hand goes wrong in a hundred small ways — full disks, no compression, no way to check it worked. Tools built for this job (pgBackRest, WAL-G) handle it all automatically: they squeeze, encrypt, verify, and clean up old backups, and they can restore to any moment with a single command. Serious setups always use a tool, never a homemade script.

✅ Do

Use pgBackRest or WAL-G to object storage with compression + encryption
Run its check/verify regularly and alert on failures
Set retention so WAL and base backups jointly cover your PITR window

❌ Don't

Hand-roll archive_command with cp/scp for production
Skip the periodic verify — an unverified backup pipeline fails silently

Gotcha: even great tooling fails silently if you never run its check/verify and never test a restore. pgBackRest can be dutifully pushing WAL to a bucket the restore role can't read, or expiring WAL your retention math got wrong — and you won't know until a real recovery. The tool automates the mechanics; it does not automate confidence. That still comes from a scheduled, timed restore drill (topic 11).

10Streaming replication & replicas▶

Scenario: your primary's server dies at 3am. With only backups, you're restoring for an hour (bad RTO). With a streaming replica already caught up, you promote it in seconds and you're back — that's the availability half of DR. But choose sync vs async carefully: it's the difference between "zero data loss" and "fast, but maybe lose the last few writes."

What

A standby continuously replays the primary's WAL stream, staying a hot, near-live copy. Async = fast, may lag; sync = primary waits for the standby to confirm (zero data loss, slower writes).

Why

Replicas give you low RTO (promote in seconds) and read scaling. They're the availability backbone — but not a substitute for backups (they copy mistakes).

How

Stand up a standby from a base backup, stream WAL, monitor lag. On failure, promote it to primary and repoint apps (topics 20–21).

A hot copy for fast failover — sync vs async

   PRIMARY ──WAL stream──► STANDBY (replays continuously, ready to promote)
      │                        │
   ASYNC:  primary acks the client FIRST, ships WAL after
           → fast writes, but a crash can lose the last few txns (RPO > 0)
   SYNC:   primary WAITS for standby to confirm, THEN acks client
           → zero data loss on failover (RPO 0), but slower writes

   promote standby → new primary in seconds (great RTO); still keep backups!

🧠

Analogy 1 — a co-pilot who can take the controls instantly: the standby flies alongside, mirroring every move, ready to take over the moment the pilot is incapacitated — that's fast recovery (RTO). Sync replication is the co-pilot confirming each instruction before you continue; async is trusting they're keeping up.

✍️

Analogy 2 — dictating to a scribe: async, you keep talking and trust the scribe to catch up (fast, but if the room clears they missed your last sentence). Sync, you pause until the scribe says "got it" before the next sentence — nothing is ever lost, but you speak slower.

Build a standby, watch the lag, promote on failure

# create a standby from the primary (writes standby config with -R)
$ pg_basebackup -h primary -D /var/lib/pgsql/data -Fp -Xs -R -P

# on the primary, require a synchronous standby for zero-data-loss writes
#   postgresql.conf:  synchronous_standby_names = 'standby1'

# watch replication health & lag (this lag is your async RPO, topic 22)
$ psql -c "SELECT client_addr, state, sync_state,
             write_lag, replay_lag FROM pg_stat_replication;"

# on primary failure, promote the standby to a read/write primary
$ pg_ctl promote        # or: SELECT pg_promote();   → new primary in seconds

🧒

In plain wordsA replica is a live twin of your database, constantly copying every change. If the main one dies, you "promote" the twin and you're back in seconds instead of restoring for an hour. The key choice: make the main one wait for the twin to confirm each write (never lose data, but a bit slower), or let it race ahead (faster, but a sudden crash might lose the last couple of writes). Twins are for uptime — you still need backups for mistakes.

✅ Do

Run standbys for low RTO; monitor replay_lag continuously
Use synchronous replication where RPO must be zero (weigh the write cost)
Keep backups too — replicas don't protect against logical errors

❌ Don't

Treat a replica as your backup — it replicates corruption and deletes
Run a single synchronous standby with no fallback — if it stalls, writes block

Gotcha: a lone synchronous standby is a hidden availability trap — because the primary waits for it to confirm every commit, if that standby gets slow or unreachable, writes on the primary stall. You traded an availability risk for a different one. The fix is quorum/multiple sync candidates (ANY 1 (s1, s2, s3)) so one slow replica can't freeze production. Sync replication protects RPO but must be designed so it doesn't sink your RTO.

11Restore drills — testing that a backup actually restores★ do this▶

Scenario: the outage is real, you reach for the backups, and… the restore fails — a missing WAL segment, an unreadable bucket, a wrong key, a corrupt archive. Backups had been "running green" for a year, but nobody ever restored one. An untested backup isn't a backup; it's a hope. The only proof is a restore you actually performed and timed.

What

A restore drill: regularly restore your backups into a scratch environment, verify the data, and time it — turning assumptions into a measured RTO and proven recoverability.

Why

Backups fail in ways only a restore reveals: gaps, bad perms, wrong keys, corrupt files, forgotten globals. And your real RTO is whatever the drill measured, not the slide.

How

Automate a periodic restore-and-validate: fetch the backup, restore to a target time, run integrity checks, record the elapsed time, alert on failure.

"Backups green" proves nothing; a restore proves everything

   backup job ✅✅✅✅✅  (a year of green checkmarks)
        │
        ▼  but has anyone RESTORED one?
   restore drill:  fetch → restore to target time → VALIDATE → time it
        ├─ row counts / checksums match?     → recoverable ✓
        ├─ how long did it take?             → your REAL RTO
        └─ anything missing (WAL, globals)?  → fix BEFORE the real outage
   schedule it (weekly/monthly) & alert if it fails — like any other test.

🧠

Analogy 1 — a fire drill, not just a fire extinguisher on the wall: an extinguisher you've never tested might be empty. The drill — actually pulling the pin — is what proves it works and teaches everyone the steps. Restoring a backup is pulling the pin.

🪂

Analogy 2 — repacking and test-jumping a parachute: a chute folded neatly in a bag looks fine; you only know it opens by jumping. A restore drill is the test jump — done on purpose, from a safe height, before your life depends on it.

Automate restore-and-validate; record the time

# scheduled DR drill (pseudo) — run monthly, alert on any failure
$ START=$(date +%s)
$ pgbackrest --stanza=shop --type=time \
    --target="$(date -d '1 hour ago' '+%Y-%m-%d %H:%M:%S+00')" \
    --pg1-path=/scratch/restore restore     # restore into a SCRATCH cluster

$ pg_ctl -D /scratch/restore start
$ psql -p 5433 -c "SELECT count(*) FROM orders;"      # validate: sane data?
$ psql -p 5433 -c "SELECT pg_is_in_recovery();"       # promoted correctly?
$ echo "RTO this drill: $(( $(date +%s) - START ))s"  # record REAL RTO
# fail the job (page on-call) if any step errors — a broken restore is an incident

🧒

In plain wordsJust because a backup ran doesn't mean it works — the only way to know is to actually rebuild the database from it and check the data is all there. So on a schedule, restore a backup into a spare place, make sure it looks right, and time how long it took. That timer is your true "how fast can we recover" number. Skipping this is how teams discover their backups are broken during the real emergency.

✅ Do

Schedule automated restore-and-validate drills (weekly/monthly)
Record the elapsed time — that's your real, defensible RTO
Validate the data (row counts, checksums), not just that the process ran

❌ Don't

Trust "backup job succeeded" as proof you can recover
Only ever test-restore during an actual disaster

Gotcha: the restore drill is the single highest-leverage, most-skipped DR practice — it's invisible when it passes and career-defining when its absence surfaces mid-outage. Even where backups are perfect, drills catch the human gaps: nobody knew the KMS key, the runbook was stale, the restore took 6 hours not 1. Test restores into a fresh environment on a schedule, or you don't have a recovery plan — you have a recovery wish.

Part III · Disaster Recovery on RDS & Aurora

12RDS automated backups & manual snapshots▶

Scenario: you're on RDS and assume "AWS handles backups." Partly true — but the defaults may keep only 7 days, automated backups are deleted when you delete the instance, and they don't leave the region. Knowing exactly what RDS backs up, for how long, and where, is the difference between a smooth restore and a nasty surprise.

What

RDS offers automated backups (daily snapshot + 5-min transaction logs → enables PITR, kept for a retention window) and manual snapshots (you take them; they persist until you delete them).

Why

Automated backups power RDS point-in-time restore; manual snapshots survive instance deletion and can be copied cross-region/account. You need both, configured deliberately.

How

Set a retention period that covers your PITR window, take manual snapshots before risky changes, and copy critical snapshots out of the region/account.

Two backup types on RDS — different lifecycles

  AUTOMATED BACKUPS                 MANUAL SNAPSHOTS
  ──────────────────────────        ────────────────────────────
  daily snapshot + 5-min tx logs    on-demand, point-in-time image
  → enables PITR within retention   → kept until YOU delete them
  retention: 0–35 days (default 7)  → copyable cross-region/account
  DELETED when you delete the DB!   → survive instance deletion

  set retention ≥ your PITR need · take a manual snap before migrations
  · copy the important ones out of region+account (topic 18)

🧠

Analogy 1 — a rolling security tape vs a printed photo you keep: automated backups are a tape that only holds the last N days and gets thrown out when you close the shop. A manual snapshot is a photo you print and file — it survives even if the shop closes, and you can mail copies elsewhere.

📸

Analogy 2 — autosave vs "Save As": automated backups are autosave (rolling, tied to the document's life). A manual snapshot is "Save As" to a permanent, named file you can move and archive — the copy that outlives the original.

Configure retention, snapshot before risk, verify

# set automated-backup retention to cover your PITR window
$ aws rds modify-db-instance --db-instance-identifier shop-prod \
    --backup-retention-period 14 --apply-immediately     # 0 = DISABLED!

# take a durable manual snapshot before a risky migration
$ aws rds create-db-snapshot --db-instance-identifier shop-prod \
    --db-snapshot-identifier shop-prod-premigration-2026-07-15

# what point-in-time can we restore to right now?
$ aws rds describe-db-instances --db-instance-identifier shop-prod \
    --query 'DBInstances[0].LatestRestorableTime'         # your current RPO edge

🧒

In plain wordsRDS does back up your database automatically, but with limits people forget: it only keeps a rolling window (often just a week), and — importantly — those automatic backups get deleted when you delete the database. So also take your own "manual snapshots" before risky changes; those stick around until you remove them and can be copied to safety. Know your retention window and set it long enough.

✅ Do

Set backup retention to cover your required PITR window (up to 35 days)
Take a manual snapshot before migrations and major changes
Copy critical manual snapshots to another region and account (topic 18)

❌ Don't

Leave retention at 0 (disabled) or assume the default covers your RPO
Forget that deleting the instance deletes its automated backups

Gotcha: deleting an RDS instance deletes its automated backups with it — teams "clean up" an old instance and unknowingly destroy the only recovery path. RDS offers a "retain automated backups" option and a final snapshot on delete; use them, and keep independent manual snapshots for anything important. Automated backups also never leave their region on their own, so a regional outage takes them too unless you've copied snapshots out.

13RDS Multi-AZ — high availability, not disaster recovery★ misunderstood▶

Scenario: "We're Multi-AZ, so we're covered for DR." This is the most common and most dangerous RDS misconception. Multi-AZ gives you automatic failover within one region if a server or AZ fails — genuinely useful — but it is not a backup, does not protect a whole-region outage, and replicates a bad DELETE to the standby instantly.

What

Multi-AZ keeps a synchronous standby in a second Availability Zone of the same region and fails over automatically (usually 60–120s) if the primary or its AZ dies.

Why

It's excellent HA (survives hardware/AZ failure with low RTO) — but people mistake it for DR. It doesn't cover region loss, human error, corruption, or accidental deletion.

How

Use it for availability, and layer real DR on top: cross-region snapshot copies/replicas for region loss, PITR/backups for logical failures.

What Multi-AZ does — and the three things it doesn't

   REGION us-east-1
   ┌──────────────────────────────────────────┐
   │  AZ-a  primary ══sync══► AZ-b  standby     │  auto-failover in ~60–120s
   └──────────────────────────────────────────┘
   ✅ survives: server crash, AZ outage           (great HA, low RTO)
   ❌ region us-east-1 goes down    → standby is in the SAME region, also down
   ❌ bad DELETE / migration         → replicated to the standby instantly
   ❌ you delete the DB              → both gone; it's not a backup
   Multi-AZ = availability. DR still needs cross-region + PITR.

🧠

Analogy 1 — a spare engine in the same plane: a second engine saves you if one fails mid-flight (HA) — but it won't help if the whole plane goes down, and it can't undo a wrong turn the pilot made. Multi-AZ is the second engine, not a parachute or a flight plan.

🏢

Analogy 2 — a backup generator in the same building: it covers a blown fuse, not the building burning down. Multi-AZ's standby lives in the same "building" (region); a regional disaster takes both. True DR keeps a copy in another city.

Multi-AZ is one setting; DR is everything around it

# enable Multi-AZ (HA within the region) — good, but not DR
$ aws rds modify-db-instance --db-instance-identifier shop-prod \
    --multi-az --apply-immediately

# DR still requires, ON TOP of Multi-AZ:
#   • a cross-region read replica or copied snapshots (region loss)  → t14,18
#   • automated backups / PITR (bad writes, corruption)              → t15
#   • an immutable, cross-account copy (ransomware, deletion)        → t5,18
$ aws rds describe-db-instances --db-instance-identifier shop-prod \
    --query 'DBInstances[0].MultiAZ'      # true ≠ "we have DR"

🧒

In plain wordsMulti-AZ keeps a second copy of your database in a nearby data center and switches to it automatically if the first one breaks — that keeps you up. But it's not a backup: it copies mistakes instantly, and if the whole region (all the nearby data centers together) goes down, the copy goes down too. It's great for uptime, but you still need separate backups and an out-of-region copy for real disasters.

✅ Do

Use Multi-AZ for in-region high availability and low failover RTO
Layer cross-region copies and PITR on top for actual DR
Say it out loud in reviews: "Multi-AZ is HA, not our DR plan"

❌ Don't

Treat Multi-AZ as a backup or as region-failure protection
Assume the sync standby saves you from a bad write — it copies it instantly

Gotcha: "Multi-AZ = DR" is the belief that gets companies caught in a regional outage with no recovery. Multi-AZ's standby is synchronous and in-region: it shares the region's fate and mirrors your mistakes. It buys availability (RTO on hardware/AZ failure), nothing more. Every RDS DR design must add a cross-region path and point-in-time backups — Multi-AZ is a feature you enable, not a plan you rely on.

14Read replicas & cross-region replicas▶

Scenario: a regional outage is your nightmare, and Multi-AZ won't save you. A cross-region read replica keeps a continuously-updated copy of your database in a different region, ready to promote into a standalone primary — turning "the region is gone" from an extinction event into a promote-and-repoint operation with minutes of RTO.

What

An RDS read replica asynchronously replicates from the primary. Placed in another region, it's a warm DR copy you can promote to a full read/write primary during a regional disaster.

Why

It's the practical answer to region loss for RDS: a live copy outside the blast radius. It also offloads reads and can seed a new region.

How

Create a cross-region replica, monitor its lag (your RPO), and on disaster promote it (it becomes independent) and repoint applications.

A warm copy outside the blast radius

   us-east-1 (primary)  ──async replication──►  us-west-2 (read replica)
        │  region outage ✗                            │  survives — different region
        │                                             ▼
        └───────────────────────────► promote replica → new PRIMARY (r/w)
                                        then repoint apps (topic 21)

   RPO = the replica's lag (async → seconds, watch it, topic 22)
   RTO = promotion time + app cutover (minutes) — much better than restore

🧠

Analogy 1 — a fully-stocked branch office in another city: if your headquarters floods, you don't rebuild from filing boxes — you promote the branch office (already staffed and current) to headquarters. The cross-region replica is that branch: warm, current, one decision away from taking over.

🚚

Analogy 2 — a delivery truck trailing a few minutes behind: the replica follows the primary a short distance back. If the lead truck crashes, the follower is nearly caught up and takes point — you lose only the small gap between them (the lag = your RPO).

Create it, watch the lag, promote on regional disaster

# create a CROSS-REGION read replica (warm DR copy)
$ aws rds create-db-instance-read-replica \
    --db-instance-identifier shop-dr-west \
    --source-db-instance-identifier arn:aws:rds:us-east-1:...:db:shop-prod \
    --region us-west-2

# monitor replica lag — this IS your RPO for a regional failover (topic 22)
$ aws cloudwatch get-metric-statistics --namespace AWS/RDS \
    --metric-name ReplicaLag --dimensions Name=DBInstanceIdentifier,Value=shop-dr-west ...

# DISASTER: promote the replica to a standalone read/write primary
$ aws rds promote-read-replica --db-instance-identifier shop-dr-west
# → now independent; repoint apps to the new endpoint (topic 21)

🧒

In plain wordsTo survive a whole region going dark, keep a live copy of your database in a different region that's always catching up to the original. If disaster strikes, you "promote" that copy to become the new main database and point your apps at it — back in minutes instead of hours. The small gap it's behind by is the little bit of data you might lose, so you watch that gap closely.

✅ Do

Keep a cross-region read replica for region-loss recovery
Monitor its replication lag — that lag is your regional RPO
Rehearse the promote + app-cutover so the real one is muscle memory

❌ Don't

Rely on same-region replicas for region-failure DR
Forget that promotion is one-way — a promoted replica won't rejoin the old primary

Gotcha: a cross-region read replica is asynchronous, so a sudden region loss can drop the last few seconds of writes (RPO = the lag at that instant) — fine for many apps, unacceptable for a ledger (use Aurora Global Database, topic 17, for tighter cross-region RPO). Also, promotion is irreversible and the replica needs enough capacity to serve full production load once promoted — a tiny under-provisioned DR replica that melts under real traffic isn't recovery.

15RDS point-in-time restore & snapshot restore▶

Scenario: a bad deploy corrupted data at 14:07. On RDS you don't hand-replay WAL — you click (or call) restore to point in time, name 14:06, and AWS spins up a brand-new instance at that moment from the automated backups. Crucially, it restores to a new instance, so your damaged one is untouched while you validate.

What

RDS point-in-time restore creates a new instance at any second within the backup-retention window (using the daily snapshot + transaction logs). Snapshot restore creates one from a specific snapshot.

Why

It's the managed equivalent of PITR (topic 8) — the way you undo logical disasters on RDS — and it always lands in a new instance, preserving the original for forensics.

How

Restore to a new instance at the target time, validate the data, then cut the app over (rename or repoint). Never overwrite the original blindly.

Managed PITR — always into a NEW instance

   shop-prod (damaged @14:07) ── stays up for forensics ──┐
                                                          │
   restore-to-point-in-time  --restore-time 14:06:00      ▼
        └──► shop-restored  (NEW instance, clean state)  ── validate ──►
                                                          repoint app (t21)
   within backup-retention window · new endpoint · original untouched
   (snapshot-restore is the same idea, from a chosen snapshot)

🧠

Analogy 1 — restoring a document to a spare copy, not over the original: you don't overwrite the file you're diagnosing — you open a fresh copy from version history at 14:06, check it's good, then switch to it. RDS PITR always makes the fresh copy for you.

🕰️

Analogy 2 — a fresh printout from the archive: instead of trying to un-spill coffee on the original, you print a clean copy from the archive dated one minute before the spill. The stained one stays on the desk in case you need to see what happened.

Restore to a new instance, validate, then cut over

# restore to a NEW instance at the second before the damage
$ aws rds restore-db-instance-to-point-in-time \
    --source-db-instance-identifier shop-prod \
    --target-db-instance-identifier shop-restored \
    --restore-time 2026-07-15T14:06:00Z        # any time in the retention window

# (or restore from a specific snapshot)
$ aws rds restore-db-instance-from-db-snapshot \
    --db-instance-identifier shop-restored \
    --db-snapshot-identifier shop-prod-premigration-2026-07-15

# validate the restored instance, THEN repoint the app (rename endpoints)
$ psql -h shop-restored... -c "SELECT count(*) FROM orders;"   # sane? cut over.

🧒

In plain wordsOn RDS, undoing a mistake is a button: "restore to this exact time." AWS builds a new database as it looked at that moment, from the automatic backups — leaving your broken one alone so you can investigate. You check the new copy looks right, then point your app at it. It's the easy, managed version of the database time-machine.

✅ Do

Restore to a new instance at just before the incident; validate before cutover
Keep the damaged instance for forensics until you've confirmed recovery
Know your window — you can only PITR within the backup-retention period

❌ Don't

Expect an "in-place rewind" — RDS restores create a new instance + endpoint
Forget to move parameter groups, security groups, and app connection strings

Gotcha: RDS PITR restores to a new instance with a new endpoint — recovery isn't done when the data is back; it's done when the app is talking to the new instance. That cutover (renaming instances, updating connection strings/secrets, moving the parameter and security groups) is where real RTO goes, and it's easy to forget under pressure. Bake the endpoint-swap steps into the runbook (topics 21, 24), because a perfect restore nobody's connected to is still an outage.

16Aurora — storage, continuous backup & Backtrack▶

Scenario: you move to Aurora PostgreSQL and the DR story changes. Storage is a distributed, self-healing layer replicated 6 ways across 3 AZs; backups are continuous to S3 with no performance hit; and some engines offer Backtrack to rewind the cluster in place in seconds. Aurora shifts much of the mechanics to AWS — but you still own the strategy.

What

Aurora separates compute from a shared distributed storage layer (6 copies / 3 AZs). Backups are continuous and automatic to S3; PITR is built in. Backtrack (where supported) rewinds the cluster to a recent time without a restore.

Why

Higher baseline durability and easier PITR, plus faster in-region recovery. Backtrack can undo a recent bad change in seconds — no new-instance restore needed.

How

Rely on continuous backup for PITR; use Backtrack for quick "oops" rewinds within its window; still copy backups cross-region/account for real DR.

Aurora moves durability into the storage layer

   Aurora cluster
     compute (writer + readers)
          │  writes log records
          ▼
     ┌──────────────────────────────────────────┐
     │  distributed storage: 6 copies / 3 AZs    │  self-healing, +continuous
     │  continuous backup to S3 (no perf hit)     │  backup → built-in PITR
     └──────────────────────────────────────────┘
   BACKTRACK: rewind the SAME cluster to e.g. 5 min ago in seconds
              (no restore, no new instance) — undo a recent bad change fast

🧠

Analogy 1 — a document with unlimited, instant version history: Aurora's continuous backup is like a doc that saves every change automatically to the cloud; Backtrack is hitting "revert to 5 minutes ago" and the same document instantly rolls back — no downloading an old copy into a new file.

🎞️

Analogy 2 — a DVR you can rewind live: instead of digging out a recording (restore), Backtrack rewinds the live broadcast in place. Great for undoing a recent mistake in seconds — but its buffer only goes back so far, so it's not your only backup.

Continuous backup, and Backtrack for fast rewinds

# PITR is continuous on Aurora — restore to any time in the window
$ aws rds restore-db-cluster-to-point-in-time \
    --db-cluster-identifier shop-restored \
    --source-db-cluster-identifier shop-aurora \
    --restore-to-time 2026-07-15T14:06:00Z

# BACKTRACK (Aurora, supported engines): rewind the SAME cluster in place
$ aws rds backtrack-db-cluster --db-cluster-identifier shop-aurora \
    --backtrack-to 2026-07-15T14:06:00Z      # seconds, no new instance
# check the window you can backtrack within:
$ aws rds describe-db-clusters --db-cluster-identifier shop-aurora \
    --query 'DBClusters[0].BacktrackWindow'

🧒

In plain wordsAurora is Postgres with a smarter storage system: it keeps six copies of your data across three data centers and backs up continuously, so it's very hard to lose. Some versions add "Backtrack" — a rewind button that snaps the same database back to a few minutes ago in seconds, perfect for undoing a fresh mistake. AWS handles more of the plumbing, but you still decide the overall plan and keep out-of-region copies.

✅ Do

Use Aurora's continuous backup for easy PITR; enable Backtrack where it fits
Still export/copy snapshots cross-region and cross-account for real DR
Know the Backtrack window — it's a short recent buffer, not full history

❌ Don't

Assume Aurora's 6-way storage removes the need for cross-region DR
Treat Backtrack as your backup — it only reaches back a limited window

Gotcha: Aurora's six-copies-across-three-AZs durability is all within one region — impressive against disk/AZ failure, but a regional outage still needs Aurora Global Database (topic 17) or cross-region snapshot copies. And Backtrack is a fixed-length rewind buffer (hours, not weeks) that only undoes changes on the same cluster — it's a fast "oops" button, not a substitute for retained, off-region backups. Aurora reduces the mechanics you manage; it doesn't reduce the strategy you must own.

17Aurora Global Database — cross-region DR▶

Scenario: your ledger needs to survive a full region loss with an RPO measured in seconds and an RTO of about a minute — tighter than an async read replica gives. Aurora Global Database replicates at the storage layer to secondary regions with typically sub-second lag, and lets you fail over to another region fast. This is the managed near-zero-RPO cross-region option.

What

An Aurora cluster spanning regions: one primary (read/write) and up to five secondary read-only regions, replicated via the storage layer with typically <1s lag. Any secondary can be promoted.

Why

It delivers low cross-region RPO (~seconds) and fast RTO (managed/planned failover ~1 min) — the strong-DR option for critical, region-resilient workloads.

How

Add secondary regions; reads serve locally. On regional disaster, fail over to a secondary (it becomes the writer) and repoint apps to the region's endpoint.

One database, many regions, sub-second replication

   PRIMARY us-east-1 (writer)
        │  storage-layer replication (typically <1s lag)
        ├──────────────► SECONDARY us-west-2 (read-only, promotable)
        └──────────────► SECONDARY eu-west-1 (read-only, promotable)

   region us-east-1 down → FAILOVER: promote a secondary to writer (~1 min)
   RPO ~ seconds · RTO ~ minute · reads served locally in each region
   the strong option when a cross-region async replica's RPO is too loose

🧠

Analogy 1 — synchronized offices worldwide sharing one ledger: every regional office sees the same book updated within a second, and any office can become headquarters on a moment's notice. If one city goes dark, another takes over the pen almost immediately, with virtually nothing lost.

🌐

Analogy 2 — a live-mirrored broadcast in several countries: the feed is replicated so tightly that if the origin studio loses power, another country's studio picks up the broadcast within a minute, and viewers barely miss a beat.

Span regions; fail over to a secondary on disaster

# create a global cluster and attach a secondary region
$ aws rds create-global-cluster --global-cluster-identifier shop-global \
    --source-db-cluster-identifier arn:aws:rds:us-east-1:...:cluster:shop-aurora
$ aws rds create-db-cluster --region us-west-2 \
    --db-cluster-identifier shop-aurora-west \
    --global-cluster-identifier shop-global --engine aurora-postgresql

# DISASTER: promote the secondary region to be the new writer
$ aws rds failover-global-cluster --global-cluster-identifier shop-global \
    --target-db-cluster-identifier arn:aws:rds:us-west-2:...:cluster:shop-aurora-west
# then repoint apps to the us-west-2 endpoint (topic 21)

🧒

In plain wordsFor the most important databases, Aurora Global Database keeps live copies in several regions, all updated within about a second of each other. If an entire region fails, you promote one of the other regions to take over in roughly a minute, losing only seconds of data. It's the premium option for things that absolutely cannot go down or lose data when a whole region disappears.

✅ Do

Use Global Database for critical workloads needing seconds-RPO across regions
Serve reads locally from secondaries; rehearse the regional failover
Automate the app repoint to the promoted region's endpoint

❌ Don't

Reach for it (and its cost) when an async replica's RPO is acceptable
Forget it still needs point-in-time backups underneath for logical errors

Gotcha: Global Database gives fantastic physical cross-region resilience but, like all replication, it faithfully copies logical disasters — a bad write reaches every region in under a second. It protects against losing a region, not against a corrupting migration; you still need continuous backups/PITR beneath it. Also distinguish managed planned failover (coordinated, ~no data loss) from unplanned/detach failover during a real region outage (fast, but you may accept the small in-flight RPO) — rehearse the unplanned path, since that's the one a real disaster triggers.

18Cross-region & cross-account copies — isolating the blast radius▶

Scenario: an attacker (or a fat-fingered admin) with access to your production account deletes the database and every snapshot in one sweep. Everything in that account shared its fate. The copy that saves you is the one in a separate account and separate region, that production's credentials can't touch. Backups must escape the blast radius of the thing they protect.

What

Copying snapshots/backups to a different region (survives region loss) and a different account (survives account compromise/deletion), ideally immutable — the concrete implementation of 3-2-1 (topic 5).

Why

Same-account, same-region backups share fate with the primary. Independence is what makes a copy survive the disaster, insider, or ransomware that took the original.

How

Automate snapshot copy to a locked-down backup account in another region, with its own KMS key and tightly scoped, ideally write-only-then-immutable access.

A copy your production credentials can't delete

   PROD account / us-east-1
     ● database        ← attacker/admin with prod access can delete
     ● snapshots       ← ...and these too (same account = same fate)

   BACKUP account / us-west-2   (different account AND region)
     ● copied snapshots ← separate credentials, own KMS key, immutable
       └─ prod's blast radius does NOT reach here → survives
   automate the copy; scope the backup account so prod can't touch it.

🧠

Analogy 1 — a safe-deposit box at a different bank: keeping your valuables and their "backup" in the same house means one burglary takes both. A box at another bank, under a different key you control separately, survives the break-in at home. The backup account is that other bank.

🔒

Analogy 2 — giving a copy to a trusted third party who can't be leaned on: a copy held by someone the attacker can't reach or coerce survives even when your own doors are kicked in. Cross-account, immutable backups are that untouchable third party.

Copy snapshots out of the region and account, immutably

# copy an RDS snapshot to another REGION (survives region loss)
$ aws rds copy-db-snapshot \
    --source-db-snapshot-identifier arn:aws:rds:us-east-1:PROD:snapshot:shop-2026-07-15 \
    --target-db-snapshot-identifier shop-dr-2026-07-15 \
    --kms-key-id alias/dr-key --region us-west-2

# share + copy into a separate BACKUP ACCOUNT (survives account compromise)
$ aws rds modify-db-snapshot-attribute --db-snapshot-identifier shop-2026-07-15 \
    --attribute-name restore --values-to-add 999988887777    # backup acct id
# then, IN the backup account, copy it to own it there (prod can't delete that copy)
# + S3 backups under Object Lock (WORM) so even the backup account can't erase them

🧒

In plain wordsIf your backups live in the same AWS account as your database, then anyone (or anything) that can delete the database can delete the backups too — in one go. So copy your important backups into a different account and a different region, locked so they can't be erased. That copy sits outside the "blast radius," so even a hacker with full access to production can't destroy your last line of defense.

✅ Do

Automate snapshot copies to a separate backup account in another region
Give the backup copies their own KMS key and immutable (Object Lock) storage
Scope access so production credentials can't delete the DR copies

❌ Don't

Keep every backup in the production account/region
Let one admin role hold delete rights over both the DB and all its backups

Gotcha: cross-region copy handles region loss but not a compromised or malicious account — if prod's credentials can reach the backups, ransomware reaches them too. True blast-radius isolation needs a separate account with independent access and, ideally, immutability (S3 Object Lock / vault lock) so nobody — not even that account's admin — can delete within the retention window. The backups an attacker can delete are exactly the ones they will.

19Encryption & KMS in DR — the key must travel too▶

Scenario: disaster strikes, you copy your encrypted snapshot to the DR region, and the restore fails: the KMS key that encrypted it is region-bound and doesn't exist in the DR region. Your data is intact but unreadable where you need it. In DR, the encryption key is as critical as the backup — a backup you can't decrypt is no backup at all.

What

RDS/Aurora backups are encrypted with KMS keys that are region-specific. A cross-region copy must be re-encrypted with a key that exists in the destination region, and the DR role must have access to it.

Why

Confidentiality is required — but a key that only lives in the dead region makes the DR copy undecryptable. The key's availability is part of your recovery path.

How

Provision a DR-region KMS key up front, re-encrypt snapshots to it on copy, and grant the DR/restore roles decrypt permission — then test a restore end to end.

Data survived; can you decrypt it where you landed?

   us-east-1: snapshot encrypted with KMS key A  (region-bound)
        │  copy to us-west-2 ── must RE-ENCRYPT with a us-west-2 key B
        ▼
   us-west-2: snapshot encrypted with key B  ✓ exists in DR region
        restore role needs kms:Decrypt on key B  ✓ granted
   ❌ forget the DR-region key → data is intact but UNREADABLE on recovery
   test the full restore, decryption included, BEFORE the real disaster.

🧠

Analogy 1 — mailing a locked safe but leaving the key at home: the safe (encrypted backup) arrives in the DR region perfectly, but the key is in the region that just burned down. You need a working key waiting at the destination — the backup and its key must both make the trip.

🗝️

Analogy 2 — a translated document you can't read without the codebook: the encrypted copy is written in a cipher; without the codebook (KMS key) present and readable by the right people at the DR site, the pages are gibberish exactly when you need them most.

Re-encrypt to a DR-region key and grant decrypt

# copy the snapshot AND re-encrypt with a key that EXISTS in the DR region
$ aws rds copy-db-snapshot \
    --source-db-snapshot-identifier arn:aws:rds:us-east-1:...:snapshot:shop-2026-07-15 \
    --target-db-snapshot-identifier shop-dr-2026-07-15 \
    --kms-key-id arn:aws:kms:us-west-2:...:key/DR-KEY \  # destination-region key
    --region us-west-2

# ensure the DR/restore role can decrypt with that key (policy/grant)
$ aws kms create-grant --key-id arn:aws:kms:us-west-2:...:key/DR-KEY \
    --grantee-principal arn:aws:iam::...:role/dr-restore --operations Decrypt
# then TEST: restore the DR snapshot in us-west-2 end to end (topic 11)

🧒

In plain wordsYour backups are locked (encrypted), which is good — but the "key" to unlock them only works in one region. If you copy a backup to another region for safety, you must also make sure a working key is waiting there and the right people can use it. Otherwise, in a disaster, you'll have your data but no way to open it — a locked box with the key in the building that just burned down.

✅ Do

Provision a KMS key in the DR region and re-encrypt snapshots to it on copy
Grant the DR/restore roles decrypt on that key; store key access in the runbook
Test a full cross-region restore including decryption

❌ Don't

Copy an encrypted snapshot cross-region without a destination-region key
Assume the restore role automatically has permission to the DR key

Gotcha: KMS keys are regional, so a snapshot encrypted with a key that only exists in the failed region is undecryptable in your DR region — the data survives, but you can't open it, which is functionally identical to losing it. This surfaces only during an actual cross-region recovery, which is exactly why the restore drill (topic 11) must run in the DR region with the DR key: it's the one place this trap gets caught before it costs you the recovery.

Part IV · Failover, Operations & Testing

20Failover & promotion — manual vs automatic▶

Scenario: the primary is unresponsive. Do you wait for it to recover, or promote a standby now? Promote too eagerly during a brief network blip and you risk split-brain (two primaries taking writes); promote too late and you're extending the outage. Failover is a judgment call plus a mechanism — and knowing which is automatic and which is yours is half the battle.

What

Failover = switching the writer role to a standby. RDS Multi-AZ fails over automatically in-region; promoting a cross-region replica or a self-managed standby is usually a deliberate action.

Why

The mechanism is easy; the decision is hard. Premature promotion causes split-brain and data divergence; hesitation costs RTO. You need clear triggers and fencing.

How

Define failover criteria in advance, fence the old primary (ensure it can't keep taking writes), promote, then repoint apps (topic 21). Prefer one clear decision-maker.

Promote decisively — but fence the old primary first

   primary unreachable?
        │  is it DOWN, or just a network blip?  ← the hard question
        ▼
   FENCE old primary (stop it taking writes) ─ prevents split-brain
        ▼
   PROMOTE standby → new primary   (RDS Multi-AZ: automatic in-region;
        ▼                            cross-region replica: manual/deliberate)
   REPOINT apps to the new writer (topic 21)
   ❌ two primaries writing at once = split-brain = diverged data = pain

🧠

Analogy 1 — handing over command of a ship: if the captain is unreachable, the first officer takes command — but only after confirming the captain truly can't act, or you get two people giving conflicting orders (split-brain). Fencing is confirming the old captain has stepped down before the new one takes the wheel.

🚦

Analogy 2 — a backup traffic cop: if the lights fail, a cop directs traffic — but if the lights flicker back on and the cop keeps directing, drivers get contradictory signals and crash. One authority at a time; make sure the other is truly off.

Promote deliberately; ensure the old primary is fenced

# RDS Multi-AZ: automatic in-region failover (AWS handles it, ~60–120s)
#   you don't trigger it; you CAN force one to test:
$ aws rds reboot-db-instance --db-instance-identifier shop-prod --force-failover

# cross-region replica / self-managed: DELIBERATE promotion
$ aws rds promote-read-replica --db-instance-identifier shop-dr-west
#   self-managed:  pg_ctl promote   /   SELECT pg_promote();

# FENCE the old primary so it can't accept writes (avoid split-brain):
#   stop it, revoke its network/security-group ingress, or block via proxy
$ aws rds stop-db-instance --db-instance-identifier shop-prod   # if reachable

🧒

In plain wordsWhen the main database seems dead, someone has to decide to switch over to the backup one. The danger: if the "dead" one wasn't really dead and both start accepting writes, they disagree and you get a mess (split-brain). So first make sure the old one is truly switched off, then promote the standby, then point the apps at it. Some switches (in-region on RDS) happen automatically; cross-region ones you trigger on purpose.

✅ Do

Define clear failover triggers and one decision-maker in advance
Fence the old primary (stop it / cut its access) before/at promotion
Rehearse promotion so the mechanics aren't new during a real outage

❌ Don't

Promote on a brief blip without fencing — split-brain diverges your data
Assume a promoted replica can simply rejoin the old primary later (it can't)

Gotcha: split-brain is the failover nightmare — the old primary wasn't dead, just unreachable, and now two databases accept conflicting writes that can never be cleanly merged. Automatic failover systems prevent it with fencing/quorum; manual promotions must fence explicitly (stop the instance, revoke its security group, or block it at the proxy) before the new primary goes live. When in doubt, making the old primary unquestionably dead is safer than risking two live writers.

21Connections through failover — endpoints, DNS & RDS Proxy▶

Scenario: you promoted the standby in 30 seconds — but the app is still hammering the dead primary's IP, because every service hard-coded the old host and DNS is cached for an hour. The database recovered; the application didn't. Getting connections to follow the failover is a first-class DR problem, not an afterthought.

What

How clients find the current writer after failover: RDS/Aurora endpoints (DNS names AWS repoints), low-TTL DNS / Route 53 failover records, and RDS Proxy (a connection layer that hides failover from the app).

Why

A perfect database failover is useless if apps keep dialing the old host. Endpoint strategy often determines your actual RTO more than the database switch itself.

How

Connect via the RDS/cluster endpoint (not a raw IP), keep DNS TTLs low, use Aurora's writer endpoint or RDS Proxy so promotion transparently redirects, and store connection details in a secret you can flip.

Don't hard-code the primary — connect through something that moves

   ❌ app → 10.0.3.7 (old primary IP)  → still dead after failover
   ✅ app → shop.cluster-xyz.rds.amazonaws.com  (endpoint AWS repoints)
   ✅ app → RDS Proxy  → proxy tracks the current writer, app never changes
   ✅ app → my-db.example.com (Route 53, low TTL, failover record)

   RTO includes: promote DB  +  connections finding the NEW writer
   cached DNS / hard-coded IPs / long TTLs quietly extend your outage.

🧠

Analogy 1 — call the front desk, not a person's direct line: if you memorize one employee's extension and they're out, your call fails. Call the front desk (the endpoint/proxy) and they route you to whoever's covering. The database can change who's "on duty" without every caller relearning the number.

📇

Analogy 2 — a forwarding address vs a fixed street number: mail a fixed address and it's undeliverable when they move. A forwarding service (DNS/proxy that updates) redirects your letters to wherever they are now — so failover just updates the forwarding, not every sender.

Connect through a movable name; flip it on failover

# ✅ apps use the ENDPOINT, never a raw IP
#   DATABASE_URL=postgres://app@shop.cluster-xyz.us-east-1.rds.amazonaws.com/shop

# RDS Proxy: apps connect to the proxy; it tracks the current writer + pools conns
$ aws rds create-db-proxy --db-proxy-name shop-proxy --engine-family POSTGRESQL ...
#   → failover is transparent to the app; also smooths connection storms

# Route 53 failover record with a LOW TTL for cross-region cutover
$ aws route53 change-resource-record-sets ...   # my-db.example.com, TTL 30
# and keep the connection string in a secret you can rotate to the new region
$ aws secretsmanager update-secret --secret-id prod/db-url --secret-string ...

🧒

In plain wordsSwitching the database over is only half the job — your apps have to notice and start talking to the new one. If they memorized the old database's exact address, they'll keep calling a dead line. So connect through a name or a proxy that can be pointed at the new database instantly, and don't let that name get "remembered" for too long. Otherwise your fast failover is wasted while apps keep dialing the wrong number.

✅ Do

Connect via RDS/cluster endpoints or RDS Proxy, never raw IPs
Keep DNS TTLs low; store the connection string in a flip-able secret
Include app repoint/DNS steps in the runbook and time them in drills

❌ Don't

Hard-code the primary's IP/hostname in app config
Forget connection-pool caching — pools may cling to the dead host until recycled

Gotcha: connection pools and cached DNS are the silent RTO killers — even after DNS updates, long-lived pooled connections keep dialing the dead primary until they're recycled or the app restarts. Set sane pool max-lifetimes, use RDS Proxy (which fails over beneath the pool), and drop stale connections on failover. Measure the app's recovery time in drills, not just the database's — the gap between them is RTO you didn't know you had.

22Replication lag = your real RPO▶

Scenario: you promised an RPO of "a few seconds" backed by an async replica. Nobody watched the lag. During a traffic spike it silently grew to four minutes — so when the region failed, you lost four minutes of orders, not a few seconds. For asynchronous replication, the current lag is your current RPO, and unmonitored lag makes your RPO a lie.

What

Replication lag = how far behind a replica is from the primary. For async replicas, whatever the lag is at the moment of disaster is exactly the data you lose — your live RPO.

Why

Lag isn't constant — it grows under load, long transactions, or network strain. An RPO target is meaningless unless you continuously measure and alert on the lag that defines it.

How

Monitor lag from Postgres (pg_stat_replication) and CloudWatch (ReplicaLag), alert when it exceeds your RPO, and investigate spikes before a disaster turns them into loss.

The lag at disaster-time is the data you lose

   primary  ●────────────────────────────►  now
   replica  ●──────────────────►  (behind by "lag")
                                 └── this gap = writes NOT yet on the replica

   region fails NOW → you lose exactly "lag" worth of writes = your RPO
   lag is dynamic: spikes under load / long txns / network → RPO drifts
   alert if lag > RPO target · a 4-min lag silently broke your "few seconds"

🧠

Analogy 1 — a note-taker falling behind in a fast meeting: if the meeting suddenly ends, you lose exactly the part the note-taker hadn't caught up on. When they're one sentence behind, you lose a sentence; when they've fallen five minutes behind, you lose five minutes. Watch how far behind they are — that's your exposure.

⏳

Analogy 2 — a live broadcast delay: the "few second delay" on live TV can stretch when the system is strained. Whatever the delay is at the instant of a cut is what viewers never saw. Unmonitored, that delay drifts — and so does your promised RPO.

Measure lag from the DB and CloudWatch; alert on the RPO threshold

# from Postgres: how far behind is the standby, right now?
$ psql -c "SELECT client_addr, state,
             write_lag, flush_lag, replay_lag FROM pg_stat_replication;"
# on the replica itself, time-based lag:
$ psql -c "SELECT now() - pg_last_xact_replay_timestamp() AS replica_lag;"

# RDS/Aurora: CloudWatch ReplicaLag / AuroraGlobalDBReplicationLag
$ aws cloudwatch put-metric-alarm --alarm-name replica-lag-over-rpo \
    --namespace AWS/RDS --metric-name ReplicaLag \
    --threshold 30 --comparison-operator GreaterThanThreshold ...  # alert > 30s

🧒

In plain wordsA live backup copy is always a little behind the original. If disaster hits, you lose exactly the part it hadn't caught up on yet — so "how far behind" is literally "how much data you'll lose." And that gap isn't fixed: under heavy load it can quietly grow from seconds to minutes. If nobody's watching it, your promise of "we only lose a few seconds" can secretly become "we lost several minutes." Watch the lag and get alerted when it grows.

✅ Do

Continuously monitor replica lag (pg_stat_replication + CloudWatch)
Alert when lag exceeds your RPO target and investigate spikes
Track lag on cross-region replicas specifically — that's your regional RPO

❌ Don't

Quote an RPO you don't measure — async lag drifts under load
Ignore a slowly-growing lag until a disaster converts it into lost data

Gotcha: a single long-running transaction or a load spike can make replica lag balloon while every dashboard still shows "replicating: healthy" — status is binary, lag is the truth. Teams monitor whether replication is up but not how far behind it is, then discover at failover that their real RPO was minutes, not seconds. Alert on the lag value against your RPO target, not just on replication being connected.

23Detecting a disaster — you can't recover what you don't notice▶

Scenario: data was silently corrupting for six hours before anyone noticed — so every backup in that window is poisoned, and your PITR target must go back further than you thought, losing far more. Recovery time starts at detection, not at failure. The faster you notice a disaster, the smaller it is.

What

The monitoring that turns a failure into an alert: instance health, replication lag (topic 22), backup/archive success, disk/WAL space, error spikes, and data corruption signals (checksum failures, anomaly checks).

Why

Undetected disasters grow — corruption spreads into backups, a stalled archiver widens your RPO gap. Fast detection bounds the blast radius and your recovery target.

How

Alarm on the DR-critical signals, enable Postgres data checksums, watch backup/WAL-archive success, and alert a human — detection is the first step of every recovery.

Recovery time starts at DETECTION, not at failure

   failure ●──────── silent for 6h ────────● noticed  ●──── recover ────►
            \___ corruption spread into backups ___/
            (the longer undetected, the further back PITR must go = more loss)

   alarm on the DR-critical signals:
     instance down · replica lag > RPO · backup/WAL archive FAILED
     disk / pg_wal filling · checksum failures · sudden error/latency spike
   detection is step ZERO of recovery — you can't fix what you can't see.

🧠

Analogy 1 — a slow leak vs a burst pipe: a burst pipe you notice instantly and mop up; a slow leak behind the wall rots the house for months before you smell it. The damage isn't just the leak — it's how long it ran unseen. Detection is the moisture sensor that catches it early.

🩺

Analogy 2 — catching an illness early: the same disease is a checkup note if caught early and a crisis if ignored for months. Monitoring is the regular vitals check that turns a quiet, spreading problem into a small, treatable one.

Alarm on DR-critical signals; enable checksums

# enable page-level checksums so silent corruption is DETECTED, not spread
$ initdb --data-checksums ...     # (or the RDS/Aurora managed equivalent)
$ psql -c "SHOW data_checksums;"  # on

# alarm on the signals that mean "a disaster is starting"
$ aws cloudwatch put-metric-alarm --alarm-name wal-archive-failing \
    --namespace ... --metric-name ...ArchiveFailures --threshold 1 ...
# also alert on: instance status, ReplicaLag > RPO, FreeStorageSpace low,
#   backup job failures, and error-rate / latency anomalies from the app
$ aws rds describe-events --source-identifier shop-prod --source-type db-instance

🧒

In plain wordsYou can't fix a disaster you haven't noticed — and the longer it goes unnoticed, the worse it gets (corrupted data quietly poisons your backups too). So set up alarms for the warning signs: the database is down, the backup copy is falling behind, a backup job failed, the disk is filling up, or the data itself is corrupting. The sooner an alarm wakes someone up, the smaller and cheaper the recovery.

✅ Do

Enable data checksums so silent corruption is caught, not replicated
Alarm on backup/WAL-archive failures, lag, disk/WAL space, and instance health
Treat a failed backup or a broken archiver as a page-worthy incident

❌ Don't

Assume "no alert" means "all good" if you never alarmed on the DR signals
Let corruption or a stalled archiver run unnoticed into your backup window

Gotcha: the most expensive disaster is the slow, silent one — corruption or a failing archiver that runs for hours before anyone notices, because by then it has spread into your backups and your clean PITR target is much further back (more data lost). The unglamorous alarms — "backup job failed," "WAL archive erroring," "checksum failure" — are the ones that catch these early. Detection latency is directly added to both your data loss and your recovery time.

24The DR runbook — turning panic into procedure▶

Scenario: 3am, the database is down, and the one person who knew how to fail over is on a plane. Nobody else knows the promote command, where the DR key lives, or who can authorize the cutover. The backups were fine — the humans weren't ready. A written, tested runbook is what makes 3am a procedure instead of a panic.

What

A DR runbook: the step-by-step recovery procedure — who decides, the exact commands to promote/restore/repoint, where keys and backups live, RPO/RTO targets, and the comms/escalation plan.

Why

Under pressure, people improvise badly and forget steps. A runbook makes recovery repeatable by anyone on call, not just the one expert — and cuts RTO.

How

Write the concrete steps, assign roles, keep it current (drills catch staleness), store it where it's reachable when systems are down, and rehearse it.

A recovery anyone on-call can run at 3am

   DR RUNBOOK  (stored OUTSIDE the systems it recovers)
   ├─ roles:      incident lead · DB operator · comms · approver
   ├─ decide:     failover criteria + who authorizes the cutover
   ├─ do:         exact commands — promote / PITR restore / repoint (t15,20,21)
   ├─ keys:       where the DR-region KMS key & backup account creds live (t19)
   ├─ verify:     data checks before declaring recovered (t11)
   ├─ comms:      status page, stakeholders, the legal/notify clock
   └─ targets:    RPO/RTO for THIS system (t1)
   kept current by drills (t25) — a stale runbook fails at 3am.

🧠

Analogy 1 — a pilot's emergency checklist: when an engine fails, pilots don't reason from first principles — they run a printed checklist, memorized through practice, with clear roles between captain and first officer. The runbook is that checklist; the calm comes from the steps existing before the emergency.

🚒

Analogy 2 — a fire evacuation plan on the wall: exits marked, a muster point named, a warden assigned. Nobody debates who does what while the alarm blares. A DR runbook is the evacuation plan for your data.

Concrete, current, reachable — and role-assigned

# dr-runbook.md — excerpt (kept in a place reachable when prod is DOWN)
$ cat dr-runbook.md
  ## roles       lead=@oncall-lead  db=@dba-oncall  comms=@sev-comms
  ## decide      promote if primary unreachable > 5m AND replica lag < RPO
  ## fence       aws rds stop-db-instance --db-instance-identifier shop-prod
  ## recover     aws rds promote-read-replica --db-instance-identifier shop-dr-west
  ## repoint     update prod/db-url secret + Route53; recycle app pools (t21)
  ## keys        DR KMS key: alias/dr-key (us-west-2); backup acct: 9999-...
  ## verify      SELECT count(*) FROM orders;  + smoke tests, THEN declare up
  ## notify      status page + stakeholders + breach clock if data exposed

🧒

In plain wordsEven perfect backups don't help if, during the emergency, nobody remembers the steps or the one expert is unreachable. A DR runbook is a clear, written "in case of disaster, do exactly this" guide: who's in charge, the exact commands to bring things back, where the keys are, and who to tell. Written down and practiced, it lets anyone on call recover calmly instead of panicking at 3am.

✅ Do

Write concrete steps and exact commands; assign clear roles and an approver
Store the runbook where it's reachable when your systems are down
Keep it current — drills (topic 25) expose stale steps and missing access

❌ Don't

Depend on one hero who "just knows how" — bus factor of one is a DR risk
Keep the only copy of the runbook inside the system it's meant to recover

Gotcha: a runbook rots the moment it's written — endpoints change, commands get new flags, people leave, access gets revoked. An untested runbook fails in small, fatal ways at 3am: the command has a typo, the linked doc moved, the on-call engineer lacks the DR-key permission. Only regular drills (topic 25) keep it true. And store it outside the failing system — a runbook that lives only in the wiki hosted on the database you're recovering is a cruel joke you'll play on yourself once.

25DR game days — rehearsing the disaster on purpose▶

Scenario: your DR plan looks great on paper. Then you run a game day — deliberately fail over to the DR region — and discover the replica was under-provisioned, the runbook's promote command changed, the on-call engineer lacked the KMS permission, and the app's connection pool clung to the dead host for ten minutes. Every one of those is far cheaper to find on a Tuesday afternoon than during a real outage.

What

A game day (DR drill / chaos exercise): a scheduled, realistic rehearsal of a disaster — fail over regions, restore from backup, run the runbook end to end — and measure the result.

Why

Plans, backups, and runbooks are all unverified until exercised. Drills convert assumptions into measured RTO/RPO and surface the human, permission, and capacity gaps that only appear under a real cutover.

How

Schedule regular drills, run the actual failover/restore (ideally in a realistic env), time it against your targets, and file every gap as a fix. Increase realism over time.

Find the gaps on a Tuesday, not during the outage

   GAME DAY: deliberately trigger the disaster & run the real runbook
   ┌────────────────────────────────────────────────────────────┐
   │ 1 fail over to DR region / restore from backup              │
   │ 2 follow the runbook exactly (no shortcuts, on-call runs it) │
   │ 3 TIME it → measured RTO/RPO vs targets (t1)                 │
   │ 4 log every gap: under-sized replica, missing perm, stale cmd│
   │ 5 fix them → re-drill                                        │
   └────────────────────────────────────────────────────────────┘
   quarterly > annually. realism increases each time. gaps get cheaper.

🧠

Analogy 1 — a full-dress fire drill, not just reading the plan: reading the evacuation map isn't the same as walking the smoky stairwell. A game day is the real walk-through — you find the jammed exit door and the missing key while it's safe, not while it's burning.

🏋️

Analogy 2 — a stress test before the marathon: athletes don't discover their limits on race day — they train under real conditions first. Game days stress your recovery under realistic load so the true race (the outage) holds no surprises.

Rehearse the real thing; measure and fix the gaps

# scheduled DR game day (quarterly) — run the ACTUAL runbook, timed
$ START=$(date +%s)
# 1) simulate the disaster (e.g. force a failover / restore in DR region)
$ aws rds failover-global-cluster --global-cluster-identifier shop-global ...
# 2) on-call follows dr-runbook.md verbatim: fence, promote, repoint, verify
# 3) measure the real numbers
$ echo "measured RTO: $(( $(date +%s) - START ))s   (target: 300s)"
# 4) capture gaps as tickets: replica too small? perm missing? DNS TTL high?
#    → fix, then schedule the next drill with more realism (real load, no heads-up)

🧒

In plain wordsThe only way to know your disaster plan works is to practice a disaster on purpose — actually switch to the backup region or restore from a backup, following your written steps, and time it. You'll always find surprises: a copy that's too small, a permission nobody has, a step that's out of date. Finding those during a planned practice run is cheap; finding them during a real 3am outage is a catastrophe. So drill regularly and fix what breaks.

✅ Do

Schedule regular game days; run the real failover/restore, timed
Have the on-call engineer (not the expert) run the runbook verbatim
File every gap as a fix and increase realism each drill

❌ Don't

Let the DR plan live only on paper, "verified" by never being run
Always let the same expert drive — you're testing the plan, not the person

Gotcha: the gaps game days expose are almost never the database itself — the backup restores fine. They're the surrounding failures: an under-provisioned DR replica that melts under real traffic, an on-call engineer without the KMS/backup-account permission, an app pool that won't let go of the dead host, a runbook command that changed. These are invisible to any amount of planning and obvious within minutes of a real drill. A DR plan you've never exercised isn't a plan — it's a hypothesis.

26Capstone — a region fails, recovered end to end★ capstone▶

Scenario: 02:11, us-east-1 is having a bad day and your primary is unreachable. This is the moment every topic in this playbook was for. Let's walk one regional disaster from alarm to "recovered," and watch each piece — RPO/RTO, replica, promotion, endpoints, keys, runbook, drills — do its job.

What

An end-to-end regional failover: detect, decide, fence, promote the cross-region copy, move connections, verify, communicate, and later recover the failed region — all driven by the runbook you rehearsed.

Why

DR isn't any single feature; it's the choreography of all of them under pressure. The capstone shows how the pieces connect into one calm procedure.

How

Follow the rehearsed runbook: detection → decision against RPO/RTO → fence old primary → promote → repoint → verify → comms → post-incident and region recovery.

One region down, walked through the whole playbook

  02:11 DETECT   alarm: primary unreachable, region impaired   (t23, t22 lag)
  02:14 DECIDE   criteria met? RPO ok (lag was <10s)? approver yes  (t1, t20)
  02:16 FENCE    ensure old primary can't take writes → no split-brain (t20)
  02:17 PROMOTE  promote cross-region replica / Global DB secondary (t14,17)
  02:19 REPOINT  flip endpoint/secret + Route53; recycle app pools  (t21)
  02:24 VERIFY   row counts + smoke tests pass → declare recovered  (t11)
  02:25 COMMS    status page + stakeholders + notify clock          (t24)
  later  RECOVER rebuild us-east-1, re-establish replication, post-mortem (t25)
  RTO ~14 min · RPO ~10s — because it was DESIGNED, rehearsed, and measured.

🧠

Analogy 1 — an emergency-room trauma call: no one improvises — the team runs a rehearsed protocol, each person owning a role, moving from stabilize to treat to debrief. The outcome looks calm precisely because every step was practiced before the patient arrived. Your regional failover is that protocol for your data.

🎼

Analogy 2 — an orchestra, not a soloist: DR is many instruments (backup, replica, keys, endpoints, runbook) playing together on cue. The capstone is the full performance — and it sounds effortless only because each section rehearsed its part.

The rehearsed sequence, executed calmly

# 02:16 fence the old primary (prevent split-brain)
$ aws rds stop-db-instance --db-instance-identifier shop-prod   # if reachable

# 02:17 promote the cross-region copy to the new writer
$ aws rds failover-global-cluster --global-cluster-identifier shop-global \
    --target-db-cluster-identifier arn:aws:rds:us-west-2:...:cluster:shop-west

# 02:19 move connections: flip the secret + DNS, recycle pools
$ aws secretsmanager update-secret --secret-id prod/db-url --secret-string "...west..."
$ aws route53 change-resource-record-sets ...        # my-db.example.com → west

# 02:24 verify BEFORE declaring recovered
$ psql -h my-db.example.com -c "SELECT count(*) FROM orders;"   # sane → up
# later: rebuild us-east-1, re-establish replication, run the post-mortem (t25)

🧒

In plain wordsHere's the whole plan in action: an alarm says a region died; you check it's really down and that your backup copy is current enough; you make sure the old one is truly off; you promote the copy in another region to be the new main; you point the apps at it; you check the data looks right and tell everyone; and later you rebuild the broken region and learn from it. It's calm and quick — about fifteen minutes, losing seconds — because every piece was set up and practiced in advance. That's disaster recovery.

✅ Do

Drive the whole failover from the rehearsed runbook, roles assigned
Verify data and app health before declaring recovered; then communicate
Run a blameless post-mortem and turn findings into fixes and new drills

❌ Don't

Improvise the sequence mid-incident, or skip fencing to "save time"
Declare victory at "database promoted" — you're done when the app is serving users

Gotcha: a regional failover that looks effortless in 14 minutes is the visible tip of months of unglamorous work — set RPO/RTO, a warm cross-region copy with watched lag, a DR-region KMS key, movable endpoints, a current runbook, and repeated drills. Skip any one and the real event stalls exactly there: no key to decrypt, no capacity on the replica, apps stuck on the dead host. DR is never a feature you switch on; it's a system you design, measure, and rehearse — so that the worst night is, remarkably, boring.