0The company & the premise
Meet NimbusPay: a payments SaaS, ~2,000 engineers, running on AWS in us-east-1 with us-west-2 as the DR region. The AWS org is tightly held — only 2 DevOps/platform admins have elevated access; the other ~1,998 engineers deploy through CI/CD and never touch the console. Their starting belief: "our account can't really be hacked — almost nobody can get in."
⚠️ Addressing the premise: "only 2 people have access, so it can't be hacked"
Tight access is genuinely excellent — it makes account compromise unlikely, and it lets us weight this whole plan toward NimbusPay's far more probable disasters (human error and region outages). But DR must treat compromise as possible anyway, because "only 2 people" is not the same as "no way in":
- An admin's long-lived access key leaks in a repo, a laptop, or a
.env— the #1 cloud breach cause. - An admin gets phished (or their SSO session token is stolen) — 2 targets is a smaller haystack, not an impossible one.
- A CI/CD role that 1,998 engineers can trigger is over-permissioned and can delete data — the "2 admins" fact is irrelevant to it.
- A malicious or coerced insider among the 2 (or anyone who can assume their role) acts deliberately.
- A supply-chain compromise (a poisoned dependency in the deploy pipeline) runs with pipeline privileges.
So the plan does two things at once: make compromise genuinely hard (Section 4), and make sure NimbusPay survives it if it happens anyway — via one immutable, cross-account backup copy the production credentials cannot delete (Sections 3–4, Runbook D). That copy costs a few dollars a month; skipping it is what turns a bad day into "pay the ransom."
1Classify — RPO/RTO per data tier
DR starts by refusing to treat all data the same. NimbusPay splits its PostgreSQL into three tiers, each with its own targets and therefore its own (very different) budget. This single table drives every later decision.
Tier 0 · critical
Tier 1 · core
Tier 2 · internal
| Tier | Data | RPO | RTO | DR pattern | Monthly cost weight |
|---|---|---|---|---|---|
| 0 | Payments ledger (money movement) | ≤ 5s | ≤ 5m | Aurora Global Database, 2 regions | $$$$ |
| 1 | Core app (users, orders, config) | ≤ 1m | ≤ 30m | RDS Multi-AZ + cross-region replica | $$$ |
| 2 | Analytics, internal tools, logs | ≤ 24h | ≤ 4h | RDS single-AZ + PITR + nightly snapshot | $ |
2The architecture — two regions, two accounts
The two ideas that shape everything: a second region (survives a regional outage) and a second, isolated account (survives a compromise of the production account). Backups land in both.
┌─ AWS ORG ──────────────────────────────────────────────────────────────┐ │ │ │ APP ACCOUNT (workloads; CI/CD deploys here) │ │ ┌──────────────── us-east-1 (PRIMARY) ────────────┐ us-west-2 (DR) │ │ │ Tier0 Aurora ledger (writer) ═══Global DB═════╪══► Aurora secondary │ │ │ Tier1 RDS core Multi-AZ (a+b) ──async repl────╪══► RDS read replica │ │ │ Tier2 RDS analytics (single-AZ) + PITR │ │ │ └────────────────────────┬────────────────────────┘ │ │ daily snapshots + continuous backup │ │ ▼ │ │ BACKUP ACCOUNT (no standing human access · separate credentials) │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ AWS Backup vault — Vault Lock (WORM, immutable) │ │ │ │ copies in us-east-1 AND us-west-2 · own KMS keys │ │ │ │ PROD credentials CANNOT delete these ← survives ransomware (Runbook D)│ │ │ └────────────────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘ region loss → fail over to us-west-2 · account compromise → restore from BACKUP ACCOUNT
The same architecture, three ways
The ASCII map above is the quick glance. Here are two more lenses: an entity diagram of how the DR objects relate, and a sequence diagram of the region-failover choreography (Runbook C) as it actually plays out.
3Backup & replication design — per tier
Now build each tier to its targets. The rule from the playbook holds throughout: replication buys RTO, backups buy RPO and undo mistakes — every tier needs both, and one immutable copy lives in the backup account.
Tier 0 — Aurora Global Database (the ledger)
A single Aurora cluster spanning us-east-1 (writer) and us-west-2 (read-only secondary), replicated at the storage layer with typically sub-second lag. Continuous backup gives PITR; a secondary region gives seconds-RPO failover.
# wrap the primary cluster in a global cluster
$ aws rds create-global-cluster --global-cluster-identifier nimbus-ledger-global \
--source-db-cluster-identifier arn:aws:rds:us-east-1:APP:cluster:nimbus-ledger
# attach a secondary region (read-only, promotable in ~1 min)
$ aws rds create-db-cluster --region us-west-2 \
--db-cluster-identifier nimbus-ledger-west \
--global-cluster-identifier nimbus-ledger-global --engine aurora-postgresql
$ aws rds create-db-instance --region us-west-2 \
--db-instance-identifier nimbus-ledger-west-1 \
--db-cluster-identifier nimbus-ledger-west --engine aurora-postgresql \
--db-instance-class db.r6g.xlarge
# watch cross-region replication lag = the ledger's real RPO (alarm if > 5s)
$ aws cloudwatch put-metric-alarm --alarm-name ledger-global-lag \
--namespace AWS/RDS --metric-name AuroraGlobalDBReplicationLag \
--threshold 5000 --comparison-operator GreaterThanThreshold ... # msTier 1 — RDS Multi-AZ + cross-region read replica (core app)
Multi-AZ gives automatic in-region failover (HA); a cross-region read replica in us-west-2 is the warm standby you promote if the whole region is lost. Automated backups (14-day retention) provide PITR for mistakes.
# in-region HA + a 14-day PITR window
$ aws rds modify-db-instance --db-instance-identifier nimbus-core \
--multi-az --backup-retention-period 14 --apply-immediately
# cross-region read replica = warm standby for a regional failover
$ aws rds create-db-instance-read-replica \
--db-instance-identifier nimbus-core-west \
--source-db-instance-identifier arn:aws:rds:us-east-1:APP:db:nimbus-core \
--region us-west-2 --db-instance-class db.r6g.xlarge # size for FULL load
# alarm on replica lag = the core app's regional RPO (alarm if > 60s)
$ aws cloudwatch put-metric-alarm --alarm-name core-replica-lag \
--namespace AWS/RDS --metric-name ReplicaLag --threshold 60 ...Tier 2 — PITR + nightly snapshot (analytics)
No standby needed. Automated backups give a PITR window; a nightly snapshot is copied out to the backup account. On disaster, restore into a new instance — hours of RTO is acceptable here.
The immutable copy — AWS Backup, cross-account, Vault Lock
This is the linchpin that survives ransomware and account compromise. A central AWS Backup plan copies every tier's recovery points into a vault in the separate backup account, in both regions, and Vault Lock (compliance mode) makes them immutable — nobody, not even the backup account's own admin, can delete them before retention expires.
# (in the BACKUP account) a locked vault — WORM, un-deletable within retention
$ aws backup create-backup-vault --backup-vault-name nimbus-dr-vault \
--encryption-key-arn arn:aws:kms:us-east-1:BACKUP:key/dr-key
$ aws backup put-backup-vault-lock-configuration \
--backup-vault-name nimbus-dr-vault \
--min-retention-days 35 --max-retention-days 3650 --changeable-for-days 3
# compliance-mode lock: after the 3-day window, the lock ITSELF can't be removed
# (org-level) a backup plan that copies each tier's recovery points
# cross-ACCOUNT (into nimbus-dr-vault) AND cross-REGION (us-west-2)
$ aws backup create-backup-plan --backup-plan file://plan.json
# rules: ledger hourly (35d) · core every 6h (35d) · analytics daily (35d)
# each rule: CopyActions → [ backup-account vault (us-east-1), (us-west-2) ]changeable-for-days grace window to catch config mistakes, then it's truly immutable.4Hardening the account — the real answer to "can it be hacked?"
NimbusPay's instinct (few people = safe) is half the answer. The other half is making the access those few people have genuinely hard to steal or misuse, and ensuring the blast radius is bounded when something slips. Here's the design that earns the confidence they already feel.
Identity — remove the things that leak
The most common cloud breach is a leaked long-lived key, so NimbusPay has none. Humans log in through IAM Identity Center (SSO) with mandatory phishing-resistant MFA and short sessions; there are no IAM users and no long-lived access keys anywhere.
# humans: federated SSO, hardware/passkey MFA, 1-hour sessions — nothing to leak
# (IAM Identity Center; permission sets, not IAM users)
# machines/CI: short-lived roles via OIDC — NO stored AWS keys in CI
# GitHub Actions assumes a role scoped to DEPLOY, with NO snapshot/db-delete
$ aws iam create-role --role-name ci-deploy \
--assume-role-policy-document file://github-oidc-trust.json
# trust: token.actions.githubusercontent.com, locked to the repo + branch
# permissions: ecs/eks deploy + read; explicitly DENY rds:Delete*, backup:Delete*
# find and kill any legacy long-lived keys (they shouldn't exist)
$ aws iam list-access-keys --user-name ANY # target: zero human/CI access keysGuardrails — make destruction structurally impossible
Even a compromised admin shouldn't be able to erase the backups. Service Control Policies (SCPs) at the org level deny the destructive actions outright — a permission no principal in the app account can be granted, no matter what.
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "DenyBackupDestruction",
"Effect": "Deny",
"Action": [
"rds:DeleteDBClusterSnapshot",
"rds:DeleteDBSnapshot",
"backup:DeleteRecoveryPoint",
"backup:DeleteBackupVault",
"backup:PutBackupVaultLockConfiguration",
"kms:ScheduleKeyDeletion",
"kms:DisableKey"
],
"Resource": "*"
}]
}
The backup account is separate and has no standing human access at all — access requires a logged, approved break-glass elevation. So even total compromise of the app account cannot reach the locked vault.
Detection & break-glass
CloudTrail (org-wide, to a locked S3 bucket) and GuardDuty watch for the signals of compromise — anomalous API calls, snapshot deletes attempts, logins from new geographies, root usage. A documented break-glass role (rarely used, heavily alarmed, MFA + approval) exists for the genuine emergency.
# org-wide audit trail to a locked bucket in the backup account
$ aws cloudtrail create-trail --name org-trail --is-organization-trail \
--s3-bucket-name nimbus-audit-locked --is-multi-region-trail
# threat detection on the API plane
$ aws guardduty create-detector --enable
# alarm on the actions that mean "someone is trying to destroy our recovery"
# any Delete* on snapshots/vaults · root login · MFA disabled · new-region API
$ aws cloudwatch put-metric-alarm --alarm-name snapshot-delete-attempt ... # → page5The runbooks — step by step
Five disasters, five procedures. Each starts with the trigger, then the exact ordered steps. In a real incident the on-call engineer opens the matching runbook and follows it verbatim — no improvising. Expand each one.
AHuman error — a bad DELETE or migration▶
Trigger: a migration or query damaged/deleted data (e.g. DELETE without a WHERE) at a known time. Replicas already copied it — so this is a backup problem (PITR), not a failover.
- Stop the bleeding. Kill the offending job; put the affected service in maintenance if writes are still corrupting data. Pin the exact time of harm from CloudTrail / the app audit log — e.g.
2026-07-15 14:07:03Z. - Restore to a NEW instance, just before the harm. Core (RDS):
aws rds restore-db-instance-to-point-in-time --source-db-instance-identifier nimbus-core --target-db-instance-identifier nimbus-core-fix --restore-time 2026-07-15T14:07:00Z. Ledger (Aurora):aws rds backtrack-db-clusterif within the Backtrack window, elserestore-db-cluster-to-point-in-timeinto a new cluster. - Validate the restored copy before touching production — row counts, checksums, a smoke test. Confirm the damage is absent and legitimate data is present.
- Cut over. Repoint the app to the restored instance (swap the endpoint in the DB-URL secret / rename instances), and recycle connection pools. Keep the damaged instance running for forensics.
- Reconcile & close. If good writes happened between the target time and the cutover, replay them from the app's event log where possible; otherwise accept that small RPO gap. Decommission the damaged instance after forensics; run a blameless post-mortem.
BAZ or instance failure (in-region)▶
Trigger: a database instance or an entire Availability Zone fails in us-east-1. This is what Multi-AZ and Aurora are built for — your job is mostly to verify, not to act.
- Let it fail over automatically. RDS Multi-AZ promotes the standby in the other AZ (~60–120s); Aurora fails to another AZ replica (~30s). No manual promotion needed.
- Verify the endpoint moved. Confirm the RDS/cluster endpoint now resolves to the healthy AZ and the database accepts writes:
psql -h nimbus-core... -c "SELECT pg_is_in_recovery();"→f(it's a writer). - Recycle clingy connections. If app pools still dial the old host, restart the app tasks or let pool max-lifetime recycle them. RDS Proxy (if in front) makes this transparent.
- Confirm the standby rebuilds. Check that a new standby/replica is being provisioned so you're not left single-AZ. No data recovery is needed — the standby was synchronous.
CFull region outage — us-east-1 is down▶
Trigger: us-east-1 is impaired region-wide; the primary and its Multi-AZ standby are both unreachable. You fail the whole stack over to us-west-2.
T+0 detect (Health + our alarms) T+8 repoint apps → us-west-2 T+2 decide + approver authorizes T+12 verify (smoke + ledger checks) T+4 fail over Aurora Global (ledger) T+13 declare recovered + comms T+6 promote RDS replica (core) later rebuild us-east-1 as secondary target: RTO ~15 min · RPO: ledger ~seconds, core ≤ ~1 min (last replica lag)
- Decide. Confirm it's regional (AWS Health dashboard), that failover criteria are met, and that the replica lag at outage time was within RPO. One named approver authorizes the cutover.
- Fail over Tier 0 (ledger).
aws rds failover-global-cluster --global-cluster-identifier nimbus-ledger-global --target-db-cluster-identifier arn:aws:rds:us-west-2:APP:cluster:nimbus-ledger-west→ the us-west-2 secondary becomes the writer. - Promote Tier 1 (core).
aws rds promote-read-replica --db-instance-identifier nimbus-core-west→ the cross-region replica becomes a standalone read/write primary. - Fence the old region. Ensure nothing in
us-east-1can still take writes when it returns (it's down now; do not let it auto-rejoin as a writer later) — prevents split-brain. - Repoint the apps. Flip the Route 53 records (low TTL) and update the DB-URL secret to the us-west-2 endpoints; recycle app connection pools. This step, not the DB promotion, is usually where the RTO goes.
- Verify, then declare. Run smoke tests and ledger balance/consistency checks. Only after they pass, update the status page and notify stakeholders (and start any regulatory notify clock).
- Recover the region later. When
us-east-1returns, rebuild it as the new secondary and re-establish replication. Do not fail back during peak load; schedule it, and treat it as its own planned change.
DAccount compromise / ransomware▶
Trigger: the app account is compromised — snapshots being deleted, GuardDuty firing, or an extortion demand. Assume everything in the app account is destroyed or untrustworthy. This is the scenario NimbusPay thought impossible — and the one the backup account exists for.
- Contain. This is a security incident first: revoke/disable the compromised identities and kill their sessions, rotate credentials, and pull in security + legal. Do not negotiate; do not trust anything in the app account.
- Reach for the untouchable copy. The Vault-Locked recovery points in the backup account are intact — production's credentials were never valid there, and compliance-mode Vault Lock means even the backup account couldn't delete them. The attacker could not reach them.
- Restore into clean infrastructure. In a fresh/clean account and region, restore each tier from the vault:
aws backup start-restore-job --recovery-point-arn arn:aws:backup:us-west-2:BACKUP:recovery-point:... --iam-role-arn ... --metadata ...— ledger, then core, then analytics. - Validate each restored database (integrity, recent-ness against your RPO), then stand the apps up against the clean environment.
- Rotate everything. Every secret, key, and credential is now suspect — rotate all of them. Rebuild the app account's infrastructure from IaC rather than "cleaning" the compromised one.
- Close the door & report. Forensics to find the entry vector (leaked key? phished admin? CI role?), fix it, and complete breach notifications per your obligations.
ESilent data corruption▶
Trigger: checksum failures or an anomaly check reveal data has been silently corrupting — and it has already replicated to standbys and into recent backups.
- Find when it started. Use logs,
data_checksumsfailures, and anomaly signals to bound the start time of corruption. Assume replicas and the most recent backups are already poisoned. - Pick a PITR target BEFORE the corruption began — not the latest backup. This is why retaining enough WAL / a long-enough PITR window matters: the clean point may be further back than you'd like.
- Restore to a new instance at that clean target and validate it's actually corruption-free (the checks that first caught it should now pass).
- Reconcile the legitimate writes made after the clean point where you can (replay from the app event log); otherwise accept the RPO gap — a known, bounded loss beats keeping corrupt data.
- Fix the cause first, then cut over. Corruption is often a bug or bad hardware; resuming without fixing it just re-corrupts the fresh copy.
data_checksums and anomaly alerts are worth the small overhead.6Keeping it real — the drill calendar
A DR plan is a hypothesis until it's exercised. NimbusPay puts these on the calendar and treats a failed drill as a real incident to fix.
| Cadence | Exercise | What it proves |
|---|---|---|
| Monthly | Restore a Tier-2 snapshot from the vault into a scratch cluster; validate; time it | Backups actually restore; measures real RTO; catches KMS/permission gaps (Runbook A/E) |
| Quarterly | Regional game day: fail Tier-1 core to us-west-2 and run Runbook C end to end | Promotion + app repoint work; measures RTO/RPO; surfaces pool/DNS issues |
| Quarterly | Aurora Global managed failover test for the ledger | Tier-0 cross-region failover works within target; app follows the writer |
| Semi-annual | Account-compromise tabletop + restore from the vault into a clean account (Runbook D) | The immutable copy is truly reachable and restorable; roles/access are right |
| Annual | Full DR audit: measured vs target RPO/RTO, SCPs, Vault Lock config, access review | The whole plan is current; the runbooks aren't stale; nobody's crept in privileges |
7Readiness scorecard
Ten yes/no questions. NimbusPay can answer "yes" to all ten — and that, not "only 2 people have access," is what actually makes them safe. Score your own setup against it.