01The system design process — a repeatable method★ start here▶
Scenario: someone says "design a system that handles a million users" and you freeze, or jump straight to boxes and arrows. Great design isn't a flash of genius — it's a method: clarify, estimate, design, then refine against the specific constraints.
What
A repeatable process: clarify requirements (functional + non-functional) → estimate scale → sketch a high-level design → deep-dive & identify bottlenecks → discuss tradeoffs.
Why
It turns an intimidating open problem into steps, keeps you from over- or under-designing, and makes your reasoning legible to others (interviews and real projects alike).
How
Start with questions, not answers. Nail down what "success" means and the numbers, then let those constraints drive the architecture — not the other way around.
1. CLARIFY what exactly are we building? for whom? constraints?
2. ESTIMATE users, QPS, storage, bandwidth (topic 2)
3. DESIGN high-level boxes: clients → API → services → data
4. DEEP-DIVE pick the hard part, detail it, find bottlenecks
5. TRADEOFFS name what you optimized for & what you gave up (topic 19)
requirements → drive → architecture (never the reverse)
# CLARIFY first — the questions that shape everything: # who uses it? read-heavy or write-heavy? how much data? # latency target? availability target? consistency needs? # scale now vs in 2 years? budget? team size? # THEN estimate (topic 2), THEN design. # a design with no stated requirements is a guess dressed as an answer.
✅ Do
- Start by clarifying requirements and constraints — ask questions first
- Let the numbers (topic 2) drive the architecture, not your favorite tech
- State what you optimized for and the tradeoffs you accepted (topic 19)
❌ Don't
- Jump to a diagram before you know the scale and goals
- Over-engineer for scale you don't have, or under-plan for scale you will
02Back-of-the-envelope estimation★ core idea▶
Scenario: "Will this fit on one database? Do we need a cache? How many servers?" You can't design without rough numbers — QPS, storage, bandwidth. Estimation turns hand-waving into "we need ~5 servers and ~2TB/year," which drives every real decision.
What
Quick order-of-magnitude math to size a system: daily active users → requests/sec (QPS), read/write ratio, storage growth, bandwidth — using round numbers and known constants.
Why
Numbers reveal what's easy vs hard: 100 QPS fits on one box; 100k QPS needs sharding and caching. Estimation tells you which problem you actually have.
How
Start from users and actions/day, divide by ~100k seconds/day for average QPS, multiply for peak, and multiply data size × volume for storage. Round aggressively.
seconds/day ≈ 86,400 ≈ 100k · 1 day ≈ 10^5 s
latency: memory ~100ns · SSD ~100µs · network RTT ~1ms · disk seek ~10ms
example: 10M DAU, each does 10 writes/day
writes/day = 100M → /100k s ≈ 1,000 writes/sec avg
peak ≈ 2–3× avg ≈ ~3,000 writes/sec
each write 1KB → 100M×1KB = 100 GB/day → ~36 TB/year
# from users → QPS → servers/storage dau = 10_000_000 writes_each = 10 writes_day = dau * writes_each # 100M qps_avg = writes_day / 100_000 # ~1,000/s (100k s/day) qps_peak = qps_avg * 3 # ~3,000/s storage_day = writes_day * 1_000 # 1KB each → 100 GB/day storage_yr = storage_day * 365 # ~36 TB/year # now decisions are obvious: cache reads, shard writes, plan 36TB+/yr
✅ Do
- Memorize a few constants (100k s/day; memory/SSD/network latencies)
- Round aggressively to powers of ten — precision isn't the point
- Compute peak (2–3× average) — systems are sized for peak, not average
❌ Don't
- Skip estimation and "just build it" — you won't know what to optimize
- Chase false precision — the order of magnitude is what drives decisions
03Scaling — vertical vs horizontal★ core idea▶
Scenario: your single server is maxing out. Do you buy a bigger server (vertical) or add more servers (horizontal)? Each is right in different situations, and the choice shapes everything downstream — statelessness, load balancing, and data partitioning.
What
Vertical scaling = a bigger machine (more CPU/RAM). Horizontal scaling = more machines sharing the load. Horizontal needs stateless services and a way to spread traffic (topic 4).
Why
Vertical is simple but has a ceiling and a single point of failure. Horizontal scales further and adds redundancy, but demands statelessness and coordination.
How
Scale up until it's uneconomical or risky, then scale out: make services stateless, put a load balancer in front, and move state to shared stores/caches.
VERTICAL (scale up): [ small ] → [ BIG server ]
simple, no code changes · but a ceiling + single point of failure
HORIZONTAL (scale out): [ LB ] → [ box ][ box ][ box ]...
near-unlimited + redundant · needs STATELESS services + LB (topic 4)
rule: scale up first (simple); scale out when you must (further, resilient)
# VERTICAL: change nothing, get a bigger instance (fast ceiling) # HORIZONTAL: requires stateless app servers — # ✗ session stored in server memory → user pinned to one box, can't spread # ✓ session in a shared store (Redis) → ANY box serves ANY request # stateless services + load balancer (topic 4) + shared state = scale out # scale DATA separately: replicas for reads, sharding for writes (topic 7)
✅ Do
- Scale vertically first — it's simple and buys real time
- Make services stateless so you can scale horizontally later
- Scale the data tier separately (replicas, sharding) — it's the harder half (topic 7)
❌ Don't
- Store session/state in app-server memory if you'll ever scale out
- Jump to horizontal complexity before a bigger box would do
04Load balancing — spreading the traffic▶
Scenario: you have five app servers — how does traffic get spread across them, and what happens when one dies? A load balancer distributes requests and routes around failures. It's the front door of every horizontally-scaled system.
What
A component that distributes incoming requests across multiple servers by a policy (round-robin, least-connections), doing health checks to skip dead ones. Operates at L4 (TCP) or L7 (HTTP).
Why
It's what makes horizontal scaling (topic 3) usable — even distribution, no single overloaded server, and automatic failover when instances die or deploy.
How
Clients hit the LB's address; it forwards to a healthy backend by policy. L7 balancers add routing, TLS termination, retries, and sticky sessions.
clients ──► [ LOAD BALANCER ] ──► server A (health ✓)
policy: ──► server B (health ✓)
round-robin / ──► server C (health ✗ → skipped)
least-conn
dead server fails health check → removed from rotation automatically
L4 = fast TCP routing · L7 = HTTP-aware (paths, TLS, retries, sticky)
# distribution policies: # round-robin → even, simple # least-connections→ better when request cost varies # IP-hash / sticky → same client → same server (session affinity) # health checks: LB pings /healthz; unhealthy backends get NO traffic # L7 (HTTP) load balancer also does: # TLS termination · path-based routing (/api → svc) · retries · rate limits # make backends STATELESS (topic 3) so any server can take any request
✅ Do
- Put a load balancer in front of any horizontally-scaled tier
- Use real health/readiness checks so traffic avoids dead/cold instances
- Pick a policy that fits (least-connections for uneven request costs)
❌ Don't
- Rely on sticky sessions to hide non-stateless servers — fix the statelessness (topic 3)
- Forget the LB itself needs redundancy — it can become the single point of failure
05Caching layers — the first performance lever★ core idea▶
Scenario: reads dwarf writes (they usually do), and the database is the bottleneck. Before sharding or rewriting, you add caches — at the CDN, the app, and in front of the DB. Caching is almost always the highest-leverage, lowest-effort scaling move.
What
Storing frequently-read data in fast layers to avoid recomputing/refetching: browser/CDN (edge), application cache (Redis/Memcached), and DB-level caches. Cache-aside is the common read pattern.
Why
Most systems are read-heavy; caching offloads the expensive tier by orders of magnitude, cutting latency and cost far more cheaply than scaling the database.
How
Cache at the layer closest to the user that's safe: static assets at the CDN, hot query results in Redis with TTLs, and always plan invalidation for when data changes.
user ─► [ CDN/edge ] static assets, cached responses (nearest, fastest)
│ miss
▼ [ app cache: Redis ] hot query results, sessions (TTL)
│ miss
▼ [ database ] the expensive source of truth
each layer absorbs load so the DB only sees the true misses
⚠ every cache adds a 2nd copy of truth → plan invalidation
# by layer, closest-to-user first: # CDN/edge: images, JS/CSS, cacheable API responses (huge win, cheap) # app (Redis): hot query results, sessions, computed views (TTL + jitter) # DB: query/plan caches, connection pools # read pattern (cache-aside): # v = cache.get(k) or (v := db.get(k), cache.set(k, v, ttl=60))[1] # INVALIDATE on write: cache.delete(k) — staleness is the price of caching # reads >> writes → caching is usually the FIRST scaling move, not the last
✅ Do
- Reach for caching early — it's usually the highest-leverage scaling move
- Cache at multiple layers (CDN → app → DB), closest to the user first
- Use TTLs (with jitter) and a clear invalidation plan for writes
❌ Don't
- Cache without an invalidation strategy — stale data bugs will follow
- Cache data that must be perfectly consistent without understanding the tradeoff (topic 10)
06Choosing a database — SQL vs NoSQL★ core idea▶
Scenario: "Should we use Postgres or Mongo? Or Cassandra? Or Redis?" The honest answer is "it depends on your access patterns" — and picking by hype instead of fit is a decision you'll live with (painfully) for years.
What
Choosing the data store by workload: relational (Postgres/MySQL) for structured data + transactions + joins; NoSQL families — document, key-value, wide-column, graph — each optimized for specific access patterns and scale.
Why
The database is the hardest thing to change later. The right fit gives you the queries, consistency, and scale you need; the wrong fit fights you forever.
How
Start from access patterns and needs (transactions? joins? scale? flexible schema?). Default to a relational DB unless a specific requirement clearly points elsewhere.
RELATIONAL (Postgres) structured, transactions, joins, strong consistency
→ default choice for most apps
DOCUMENT (Mongo) flexible/nested schema, per-document ops
KEY-VALUE (Redis/Dynamo) simple lookups, caching, huge scale, low latency
WIDE-COLUMN (Cassandra) massive writes, time-series, tunable consistency
GRAPH (Neo4j) relationships/traversals (social, recommendations)
choose by ACCESS PATTERN + consistency + scale needs
# default: relational (Postgres) — it's flexible, transactional, and # scales further than people think (it also does JSON, topic in PG playbook) # choose NoSQL when a SPECIFIC need dominates: # need extreme write scale + tunable consistency → wide-column (Cassandra) # simple key lookups at huge scale / cache → key-value (Redis/Dynamo) # deeply nested, schema-flexible documents → document (Mongo) # relationship traversals are the core query → graph (Neo4j) # ⚠ you often end up POLYGLOT: Postgres + Redis + a search index
✅ Do
- Start from access patterns, consistency, and scale needs
- Default to a relational DB unless a specific requirement points elsewhere
- Expect to be polyglot — e.g. Postgres for truth + Redis for cache + a search index
❌ Don't
- Pick a database by hype or résumé-driven development
- Assume "NoSQL scales, SQL doesn't" — modern relational DBs scale far
07Partitioning & replication — scaling the data▶
Scenario: your data no longer fits or serves fast enough on one database. Two tools: replication (copies for read scale & durability) and partitioning/sharding (split data across nodes for write scale). Most large systems use both — and both have sharp edges.
What
Replication keeps copies (leader-follower or quorum) for durability and read scaling. Partitioning/sharding splits data by a key across nodes to scale writes and storage beyond one machine.
Why
Replicas absorb reads and survive node loss; shards let writes and data exceed a single box. Together they're how the data tier scales — the usual real bottleneck (topic 3).
How
Route reads to replicas (mind lag); pick a shard key (hash for even spread, range for scans) that avoids hot spots. Cross-shard queries and rebalancing are the hard parts.
REPLICATION (read scale + durability):
writes → LEADER ──log──► FOLLOWER (reads) ──► FOLLOWER (reads)
replicas lag → a just-written read may be stale (topic 10)
SHARDING (write scale + storage):
hash(user_id) → shard 0 | shard 1 | shard 2 (split by key)
even spread, but cross-shard joins & hot shards are painful
big systems: sharded AND each shard replicated
# REPLICATION — reads to replicas, writes to leader:
def query(sql, write=False):
return (LEADER if write else pick(REPLICAS)).execute(sql)
# beware replication lag → read-your-writes may need the leader (topic 10)
# SHARDING — route by a well-chosen key:
def shard(user_id): return SHARDS[hash(user_id) % len(SHARDS)]
# hash = even spread (no range scans) · range = scans (risk hot ranges)
# ⚠ the shard key is nearly permanent — choose from access patterns✅ Do
- Use replicas for read scaling and durability; handle replication lag (topic 10)
- Choose a shard key that spreads load evenly and matches your queries
- Delay sharding until you must — it's a big, hard-to-reverse jump
❌ Don't
- Shard on a skewed key (country, status) — you'll create hot shards
- Design flows needing frequent cross-shard joins/transactions — they're painful
08The API layer — REST, gRPC, GraphQL & gateways▶
Scenario: clients and services need to talk. Do you expose REST, gRPC, or GraphQL? And how do you handle auth, rate limiting, and routing without duplicating them in every service? The API layer (and an API gateway) is the contract and the front door.
What
The interface between clients and your system: REST (resources over HTTP), gRPC (fast binary RPC, service-to-service), GraphQL (client-specified queries). An API gateway centralizes cross-cutting concerns.
Why
The API is a long-lived contract — hard to change once clients depend on it. The right style fits the use case; a gateway avoids re-implementing auth/rate-limiting/routing everywhere.
How
REST for public/simple CRUD; gRPC for internal high-performance calls; GraphQL when varied clients need flexible shapes. Put a gateway in front for auth, rate limiting, routing, and TLS.
clients ─► [ API GATEWAY ] auth · rate limit · routing · TLS · logging
│ (one place for cross-cutting concerns)
┌──────────┼───────────┐
▼ ▼ ▼
REST GraphQL gRPC (internal service↔service, fast binary)
REST: simple, cacheable, public GraphQL: client picks fields
# choose by use case: # REST → public APIs, simple CRUD, cacheable, everyone understands it # gRPC → internal service↔service, high throughput, typed contracts # GraphQL → many varied clients needing different field shapes (avoids # over/under-fetching), at the cost of caching & complexity # API GATEWAY (one front door) handles cross-cutting concerns ONCE: # auth · rate limiting · routing · TLS termination · request logging # the API is a CONTRACT → version it; breaking changes break clients
✅ Do
- Choose the API style by use case (REST public, gRPC internal, GraphQL flexible)
- Put a gateway in front for auth, rate limiting, routing, and TLS — once
- Version your API and treat it as a long-lived contract
❌ Don't
- Make breaking changes to a public API without versioning — clients shatter
- Re-implement auth/rate-limiting in every service instead of at the gateway
09Async & queues at the design level▶
Scenario: a user uploads a video and your request blocks for two minutes transcoding it. Instead, accept the upload, drop a job on a queue, and return immediately — workers process it later. Moving slow/spiky work off the request path is a core design move.
What
Using message queues/streams to do work asynchronously — decoupling producers from consumers so slow, spiky, or non-urgent work happens off the request path.
Why
It keeps user-facing requests fast, absorbs traffic spikes (the queue buffers), and decouples services so they fail and scale independently.
How
Producer enqueues a job/event; workers consume at their own rate; the queue buffers bursts. Add retries, dead-letter queues, and idempotent consumers.
SYNC: upload → [ transcode 2 min ] → respond user waits, times out ✗
ASYNC: upload → enqueue job → respond "processing" (instant) ✓
[ QUEUE ] buffers bursts
│ workers pull at their own pace
▼
[ worker ][ worker ] scale independently
the queue absorbs spikes; workers catch up; user isn't blocked
# request path stays fast:
def upload(video):
id = store_raw(video)
queue.enqueue("transcode", {"id": id}) # hand off & return
return {"status": "processing", "id": id} # instant response
# workers (separate, scalable) do the slow work:
# pull job → transcode → mark done (idempotent, with retries + DLQ)
# benefits: fast responses · spike buffering · independent scaling
# use async for: media processing, emails, exports, webhooks, analytics✅ Do
- Move slow, spiky, or non-urgent work to a queue + workers
- Make consumers idempotent and add retries + a dead-letter queue
- Return fast to the user; report progress/status for async jobs
❌ Don't
- Block user requests on slow work that could be async
- Forget queues add eventual consistency and new failure modes to reason about
10Consistency vs availability — the design decision★ core idea▶
Scenario: during a network hiccup, should your system show possibly-stale data (stay up) or refuse to answer until it's sure it's correct (stay consistent)? This isn't a purely technical question — it's a business decision you make per feature, and it shapes the whole design.
What
The CAP/PACELC trade-off applied as a design choice: under a partition you pick consistency (refuse stale) or availability (serve, maybe stale); even without a partition, you trade latency vs consistency.
Why
Different features have different costs of being wrong vs being down. A bank balance leans consistent; a like-count or feed leans available. The choice drives your data and caching design.
How
Decide per use case by the cost of stale data vs the cost of downtime. Pick stores and replication modes accordingly; often tune consistency per operation.
during a network partition, per feature:
money / inventory / uniqueness → CONSISTENT (refuse if unsure)
likes / feed / recommendations → AVAILABLE (serve, maybe stale)
even with NO partition (PACELC "else"): Latency vs Consistency
strong reads = slower/coordinated · eventual reads = fast/maybe stale
it's a BUSINESS decision, not just a technical one
# decide with a simple question per feature: # what's worse here — showing STALE data, or being DOWN? # CONSISTENT (CP): payments, inventory, account balances, uniqueness # → strongly-consistent store; refuse rather than serve wrong # AVAILABLE (AP): feeds, likes, view counts, recommendations # → eventually-consistent store; always answer, reconcile later # you can MIX: strong for the ledger, eventual for the activity feed # (full theory: CAP/PACELC in the Distributed Systems playbook)
✅ Do
- Decide per feature by the cost of stale data vs the cost of downtime
- Use strong consistency for money, inventory, and uniqueness
- Accept eventual consistency (and design for it) where availability matters more
❌ Don't
- Demand strong consistency everywhere — you pay in latency and availability
- Treat this as purely technical — bring the business into the trade-off
11A worked design — a URL shortener★ worked example▶
Scenario: let's run the whole process on one classic problem — design a URL shortener (like bit.ly). It's small enough to hold in your head and touches everything: estimation, the read/write skew, key generation, caching, and the tradeoffs. This is Part I assembled.
What
Applying the method (topic 1) end to end: clarify → estimate → design the write path (create short code) and the read path (redirect) → cache → name the tradeoffs.
Why
A worked example shows how the pieces combine: a read-heavy system where estimation points straight at caching, and key generation is the interesting core decision.
How
Generate a unique short code, store code→URL, and on read look up (cache first) and 301/302 redirect. Reads vastly outnumber writes, so optimize the read path hard.
estimate: 100M new links/mo → ~40 writes/s · reads ~100× → ~4,000 reads/s
reads >> writes → CACHE the read path hard
WRITE: POST /shorten → generate code (base62 of an id / hash) →
store code→url in DB
READ: GET /abc123 → cache.get? → else DB → 301 redirect (+ warm cache)
▲ 99%+ hits here → DB barely touched
# 1) CLARIFY: custom aliases? expiry? analytics? → shapes the schema # 2) ESTIMATE: read-heavy (~100:1) → caching is the main lever # 3) KEY GEN (the interesting core): # a) counter → base62 encode (short, sequential — but guessable/leaky) # b) hash(url) + collision check (dedupes identical urls) # c) random base62, check-and-insert (unguessable; retry on collision) # 4) READ PATH: cache.get(code) or db.get(code); 301 redirect; warm cache # 5) TRADEOFFS: 301 (cacheable, loses analytics) vs 302 (counts every hit) # consistency: a brand-new link should resolve immediately (topic 10)
✅ Do
- Run the full method even on "small" problems — it builds the habit
- Let the read/write skew point you to the main lever (here, caching)
- Make the interesting core decision explicit (here, key generation) and state tradeoffs
❌ Don't
- Jump to a diagram before estimating — you'd miss that it's read-dominated
- Hand-wave the hard part (unique code generation) — that's the real design
12Non-functional requirements★ core idea▶
Scenario: two designs both "work," but one must answer in 50ms and never lose data, while the other can be slow and occasionally drop a request. The difference is non-functional requirements — the "-ilities" (latency, availability, durability, scalability) that shape the architecture as much as features do.
What
Requirements about how well the system behaves, not what it does: latency, throughput, availability, durability, scalability, security, cost, maintainability.
Why
Features tell you what to build; NFRs tell you how to build it. "99.99% available, p99 < 100ms, never lose a payment" drives every architectural choice.
How
Make them explicit and measurable up front. Each NFR has a cost — you can't max them all — so prioritize by what the business actually needs.
LATENCY how fast? (p50/p99 targets)
AVAILABILITY how often up? (99.9% = ~9h/yr down · 99.99% = ~52min/yr)
DURABILITY never lose data? (payments: yes · analytics: maybe)
SCALABILITY grows with load?
SECURITY · COST · MAINTAINABILITY
you CANNOT max them all → prioritize by business need
# make NFRs explicit and testable — vague ones are useless: # ✗ "fast" → ✓ "p99 read latency < 100ms" # ✗ "reliable" → ✓ "99.95% availability; zero payment data loss" # ✗ "scalable" → ✓ "handle 10x current traffic without redesign" # each has a COST; you're trading among them: # more availability → more redundancy → more $$ # stronger durability → sync replication → higher latency (topic 10) # prioritize by what the BUSINESS needs, not by what sounds impressive
✅ Do
- Define NFRs explicitly and measurably (p99 < X, 99.9% up, zero data loss)
- Prioritize — you can't maximize latency, availability, durability, and cost at once
- Let the NFRs drive architectural choices (replication mode, caching, redundancy)
❌ Don't
- Leave NFRs vague ("fast," "reliable") — they can't guide design or be verified
- Assume you can have every "-ility" maxed — each one costs
13SLIs, SLOs, SLAs & error budgets★ core idea▶
Scenario: "Is the system reliable enough?" is unanswerable without numbers. SLIs/SLOs turn reliability into measurable targets, and the error budget gives you a principled way to balance shipping features against staying up — ending the "devs want to ship, ops want to freeze" war.
What
SLI = a measured indicator (e.g. % of requests < 200ms). SLO = your target for it (99.9%). SLA = a contractual promise to customers (with penalties). Error budget = 1 − SLO, the allowed unreliability.
Why
They make reliability a measurable, negotiable engineering target instead of a vibe — and the error budget aligns feature velocity with stability using data, not opinions.
How
Pick a few user-centric SLIs, set SLOs, and track the error budget. Budget left → ship features. Budget spent → freeze features and fix reliability.
SLI measured: 99.95% of requests < 200ms this month
SLO target: 99.9% (internal goal)
SLA promise: 99.5% (contract with customers; penalties if missed)
ERROR BUDGET = 100% − SLO = 0.1% allowed failure (~43 min/month)
budget remaining → SHIP features
budget exhausted → FREEZE features, fix reliability
# SLI: pick user-centric signals (availability, latency, correctness) # sli = good_requests / total_requests # SLO: the target slo = 0.999 # 99.9% error_budget = 1 - slo # 0.1% of requests may fail # monthly budget in minutes ≈ 30d * 24h * 60m * (1-slo) ≈ 43 min # POLICY: # budget remaining → ship features, take reasonable risks # budget exhausted → feature freeze, invest in reliability # 100% reliability is the WRONG target — it's infinitely expensive & blocks change
✅ Do
- Pick a few user-centric SLIs; set realistic SLOs; track the error budget
- Use the error budget to decide: budget left → ship; budget gone → stabilize
- Keep SLAs (contractual) looser than internal SLOs — leave headroom
❌ Don't
- Target 100% reliability — it's infinitely expensive and blocks all change
- Set SLOs on metrics users don't feel (internal CPU%) instead of their experience
14Observability — you can't fix what you can't see▶
Scenario: production is "slow" and you have no idea why — no metrics on latency, no traces across services, no useful logs. Observability is the difference between debugging blind and knowing exactly where the problem is. As a lead, it's non-negotiable infrastructure.
What
The three pillars: metrics (numbers over time — latency, error rate, throughput), logs (discrete events with context), and traces (one request across services) — plus alerts on what matters.
Why
You cannot operate, debug, or meet SLOs (topic 13) without visibility. Observability turns "it's broken somewhere" into "payment latency spiked at 2:14 on shard 3."
How
Instrument the four golden signals (latency, traffic, errors, saturation), propagate a trace id everywhere, log with structure, and alert on user-facing symptoms, not every internal blip.
METRICS latency · traffic · errors · saturation (the 4 golden signals)
LOGS structured events + a request id for correlation
TRACES one request's path across services (find the slow hop)
▼
ALERTS on user-facing SYMPTOMS (SLO burn), not every CPU twitch
goal: "it's slow" → "payment p99 spiked at 2:14 on shard 3"
# the FOUR GOLDEN SIGNALS (per service): # latency (p50/p99 of requests) · traffic (req/s) # errors (error rate) · saturation (how full — CPU/mem/queue) # TRACES: propagate a trace_id through every hop (find the slow service) # LOGS: structured JSON with trace_id → filter one request's whole story # ALERTS: on USER-FACING symptoms (SLO burn rate, error spike), NOT on # "CPU is 80%" (which may be totally fine). Alert fatigue kills response.
✅ Do
- Instrument the four golden signals (latency, traffic, errors, saturation)
- Propagate trace ids and log with structure so you can correlate
- Alert on user-facing symptoms / SLO burn, not every internal metric
❌ Don't
- Wait for an outage to add observability — build it in from the start
- Alert on everything — alert fatigue makes on-call ignore the one that matters
15Designing for failure & blast radius★ core idea▶
Scenario: one dependency goes down and takes your entire product with it. Everything fails eventually — the question is how much fails when it does. Designing for failure means containing the blast radius so one broken part degrades gracefully instead of causing a total outage.
What
Assuming components will fail and designing so failures are contained and graceful: redundancy, isolation (bulkheads), timeouts/circuit breakers, and graceful degradation instead of total collapse.
Why
At scale, something is always broken. A design that assumes everything works cascades one failure into an outage; a resilient one loses a feature, not the product.
How
Add redundancy (no single point of failure), isolate failures (bulkheads, cells), bound calls (timeouts, breakers), degrade gracefully, and limit the blast radius of any one fault.
✗ fragile: recs-service down → WHOLE product 500s (blast radius = all)
✓ resilient:
• redundancy: no single point of failure (LB, replicas, multi-AZ)
• isolation: bulkheads/cells → one tenant's spike can't sink others
• timeouts + circuit breakers → a slow dep fails fast, not cascades
• degrade: recs down → hide recs, rest of page works
goal: one failure = a small crater, not the whole map
# REDUNDANCY: no single point of failure (redundant LB, replicas, multi-AZ) # ISOLATION (bulkheads/cells): separate resource pools/tenants so one # overloaded part can't starve the rest # BOUND every dependency call: timeout + retry(+jitter) + circuit breaker # → a slow/dead dependency fails FAST instead of cascading (see distsys) # DEGRADE gracefully: optional feature down → hide it, keep the core # ASK of every component: "if this dies, what else dies?" → shrink that set
✅ Do
- Assume every component will fail; ask "if this dies, what else dies?"
- Add redundancy, isolation (bulkheads/cells), timeouts, and circuit breakers
- Degrade gracefully — lose a feature, not the whole product
❌ Don't
- Design assuming dependencies are always up — they aren't
- Let a non-critical dependency's failure cascade into a total outage
16Capacity planning & cost▶
Scenario: you either run out of capacity during a spike (outage) or massively over-provision and burn money (waste). As a lead, "how much do we need, and what does it cost?" is a question you own — and the cloud makes it easy to accidentally spend a fortune.
What
Planning how much compute/storage/bandwidth you need (from estimation, topic 2) with headroom for peaks and growth — and understanding and controlling the cost of those resources.
Why
Under-provisioning causes outages; over-provisioning wastes money. And cloud cost is now an engineering concern — inefficient designs cost real dollars, forever.
How
Size for peak + headroom; autoscale where possible; know your cost drivers (compute, egress, storage tiers); and treat efficiency as a design goal, not an afterthought.
under-provisioned: peak load > capacity → OUTAGE (and lost revenue) over-provisioned: capacity >> load → WASTE (paying for idle) right: capacity = peak × headroom (e.g. 1.5×), autoscale the rest cost drivers (cloud): compute · DATA EGRESS (sneaky!) · storage tier · ops efficiency IS a feature — a wasteful design costs money every hour, forever
# from estimation (topic 2): peak QPS, storage/yr, bandwidth # capacity = peak_load * headroom(≈1.3–2x); AUTOSCALE the variable part # COST DRIVERS people forget: # data EGRESS (cross-region / out to internet) — often the biggest bill! # idle over-provisioned instances · verbose logs/metrics retention # chatty cross-service calls · un-cached reads hammering the DB # make cost VISIBLE (per-team dashboards) and set budgets/alerts # efficiency is a design constraint — a 2x wasteful design = 2x the bill, forever
✅ Do
- Size for peak plus headroom; autoscale the variable portion
- Know your real cost drivers — data egress is often the sneaky big one
- Make cost visible (dashboards, budgets, alerts) and treat efficiency as design
❌ Don't
- Size for average load — a peak spike then causes an outage (topic 2)
- Ignore cloud cost until the bill arrives — inefficiency compounds every hour
17Security & privacy by design★ core idea▶
Scenario: security bolted on at the end is expensive and leaky; a single breach can end a company. As a lead, you bake security and privacy into the design from day one — least privilege, defense in depth, encryption, and collecting only the data you truly need.
What
Designing security and privacy in from the start: least privilege, defense in depth, encryption (in transit & at rest), input validation, secrets management, and data minimization (collect/keep only what you need).
Why
Breaches are existential — financially and reputationally — and privacy is increasingly a legal requirement (GDPR, etc.). Retrofitting security is far costlier and leakier than designing it in.
How
Grant minimal permissions, layer defenses so one failure isn't fatal, encrypt everywhere, validate all input, manage secrets properly, and minimize the sensitive data you hold.
LEAST PRIVILEGE every component gets the MINIMUM access it needs
DEFENSE IN DEPTH many layers → one breach ≠ full compromise
[ WAF ][ auth ][ authz ][ input validation ][ encryption ][ audit ]
ENCRYPT in transit (TLS) AND at rest
DATA MINIMIZATION don't collect/keep data you don't need
(data you don't hold can't be breached or subpoenaed)
# LEAST PRIVILEGE: each service/role gets only the access it needs # (a read-only report job should NOT have write/delete on the DB) # DEFENSE IN DEPTH: WAF + authN + authZ + input validation + encryption + audit # → assume any single layer can fail; no layer is the only defense # ENCRYPT: TLS in transit, encryption at rest, secrets in a vault (not code/git) # VALIDATE all untrusted input; parameterize queries (no injection) # DATA MINIMIZATION: collect only what you need; set retention + deletion # → the cheapest data to protect is the data you never stored
✅ Do
- Apply least privilege and defense in depth from the start
- Encrypt in transit and at rest; keep secrets in a vault, never in code/git
- Minimize data collected/retained — set retention and deletion policies
❌ Don't
- Bolt security on at the end — it's costlier and leakier
- Hoard sensitive data "just in case" — it's pure liability (breach + legal)
18Migrations & evolving systems safely★ core idea▶
Scenario: you need to change the database schema, swap a service, or move to a new system — while it's live and serving traffic, with no downtime and no data loss. Big changes are where systems (and careers) break. Doing them safely and incrementally is a defining senior skill.
What
Evolving a running system without breaking it: backward-compatible changes, expand-and-contract (dual-write/dual-read), gradual rollout, and always a rollback path.
Why
You rarely get to stop the world and rewrite. Unsafe migrations cause outages and data loss; incremental, reversible ones let you change anything without downtime.
How
Use the expand → migrate → contract pattern: add the new alongside the old (compatible), migrate data/traffic gradually, verify, then remove the old. Roll out behind flags; keep rollback ready.
✗ big-bang: switch everything at once → if wrong, total outage, no undo
✓ EXPAND: add the new column/service ALONGSIDE the old (compatible)
MIGRATE: dual-write, backfill, dual-read; shift traffic gradually
verify at each step; roll back instantly if wrong
CONTRACT: once fully on the new path, remove the old
every step is small, reversible, and observable
# goal: change a live system with NO downtime and instant rollback # EXPAND — add the new, keep the old working (backward compatible): # add new_column (nullable); deploy code that writes BOTH old & new # MIGRATE — move data & traffic gradually: # backfill new_column from old; start reading new (fall back to old); # roll out behind a feature FLAG to 1% → 10% → 100%, watching metrics # CONTRACT — once fully migrated & verified: # stop writing old; drop the old column/service # at EVERY step: reversible + observable. Never a one-shot cutover.
✅ Do
- Use expand → migrate → contract; make each step backward-compatible and reversible
- Roll out behind feature flags gradually (1% → 100%), watching metrics
- Always have a rollback path and verify data at each step
❌ Don't
- Do a big-bang cutover of a live system — one mistake = total outage, no undo
- Make a breaking schema/API change without a compatible transition (topic 8)
19Trade-off analysis & decision records (ADRs)★ core idea▶
Scenario: six months from now, someone asks "why did we choose Kafka over SQS?" and nobody remembers — so the decision gets re-litigated, or worse, reversed for bad reasons. Every real design is a set of trade-offs, and writing them down (an ADR) is how good decisions endure.
What
Explicitly weighing options against your requirements — there's no "best," only "best for these constraints" — and recording the choice, context, and rejected alternatives in an Architecture Decision Record (ADR).
Why
Every choice has costs; naming them prevents cargo-culting and lets others evaluate the reasoning. ADRs preserve the why, so future engineers don't repeat debates or undo good calls blindly.
How
List real options, score them against your prioritized requirements, pick with eyes open, and write a short ADR: context, decision, alternatives considered, consequences.
no "best" — only "best for THESE requirements":
option A: simpler, cheaper, less scalable
option B: complex, pricier, scales further
→ score against YOUR prioritized NFRs (topic 12) → choose with eyes open
ADR (Architecture Decision Record):
CONTEXT · DECISION · ALTERNATIVES considered · CONSEQUENCES (good & bad)
preserves the WHY for the engineer who asks in 6 months
# ADR-014: Use SQS (not Kafka) for the notification queue # # STATUS: accepted (2026-02) # CONTEXT: need a simple work queue; ~1k msgs/s; small team; no replay need # DECISION: use SQS. # ALTERNATIVES CONSIDERED: # - Kafka: more powerful (replay, high throughput) but heavy ops burden # for our scale & team → rejected (over-engineered for need) # - Redis lists: cheap but no managed durability/DLQ → rejected # CONSEQUENCES: # + managed, simple, cheap, good enough for current + 10x scale # - if we later need event replay/streaming, revisit (see topic 12 NFRs)
✅ Do
- Frame every choice as a trade-off against your prioritized requirements
- Write short ADRs: context, decision, alternatives, consequences
- Record the why and the rejected options, not just the final pick
❌ Don't
- Present a choice as "the best" with no trade-offs — that's a red flag
- Leave decisions undocumented — they'll be re-litigated or blindly reversed
20Build vs buy▶
Scenario: you need auth, or search, or a payment system. Do you build it or adopt an existing service/library? Engineers love to build; the business often needs you to buy. Getting this call right saves months and keeps the team focused on what actually differentiates the product.
What
Deciding whether to build a capability in-house or adopt an existing solution (SaaS, open source, managed service) — weighing control and fit against time, cost, and maintenance burden.
Why
Building non-differentiating infrastructure wastes your scarcest resource — engineering time — and adds permanent maintenance. Buying frees the team to build what actually makes the product special.
How
Ask: is this a core differentiator? Build it. Is it a solved commodity (auth, email, payments)? Buy it. Factor in total cost of ownership, not just the sticker price.
is it a CORE DIFFERENTIATOR (your product's moat)? → BUILD
is it a solved COMMODITY (auth, payments, email, search)? → BUY
BUILD cost = dev time + FOREVER maintenance + opportunity cost
BUY cost = license + integration + some lock-in
→ compare TOTAL cost of ownership, not just the sticker price
engineers over-value building; the business needs focus on the moat
# the core question: does this DIFFERENTIATE us? # YES (our moat) → BUILD (control, fit, it IS the product) # NO (commodity) → BUY (auth, payments, email, search, monitoring...) # TOTAL COST OF OWNERSHIP of building is underestimated: # initial dev + FOREVER maintenance + on-call + opportunity cost # (every hour building undifferentiated infra is an hour NOT on the moat) # buying costs: license + integration + some lock-in — usually far cheaper # ⚠ engineers systematically over-value 'build' (it's fun) — correct for it
✅ Do
- Build what differentiates your product; buy commodity capabilities
- Weigh total cost of ownership — building means maintaining forever
- Correct for the engineer's bias toward building (it's the fun option)
❌ Don't
- Build undifferentiated infra (auth, email, payments) from scratch without strong reason
- Compare only sticker prices — the maintenance and opportunity cost dominate
21Estimation & risk in planning▶
Scenario: "How long will this take?" You say "two weeks," it takes two months, and trust erodes. Estimating engineering work and surfacing risk honestly is a core leadership skill — not to be perfectly right, but to plan realistically and communicate uncertainty.
What
Turning uncertain work into useful plans: breaking work down, estimating in ranges (not false-precise points), identifying risks and unknowns early, and communicating confidence honestly.
Why
Bad estimates blow schedules, erode trust, and hide risk until it's too late. Honest, range-based estimates with surfaced risks let the business plan and prioritize realistically.
How
Decompose into small pieces, estimate ranges, add buffer for unknowns, flag the risky/unknown parts, and prototype the scariest thing first to shrink uncertainty.
✗ "it'll take 2 weeks" (false precision → broken promise) ✓ "2–4 weeks; the risk is the payment integration (unknown)" DECOMPOSE big → small pieces (small things estimate better) RANGE low–high, not a single number (uncertainty is real) RISK name the unknowns; PROTOTYPE the scariest part FIRST the goal isn't to be right — it's to plan realistically & flag risk
# DECOMPOSE: break the work into small, estimable pieces # ESTIMATE in RANGES with confidence: # "2–4 weeks (60% confident); worst case 6 if payments is hard" # IDENTIFY RISK: list unknowns/dependencies explicitly # → the estimate's spread comes from the UNKNOWNS, not the known work # DE-RISK EARLY: prototype/spike the scariest part FIRST # → shrinks the range fast; find the disaster in week 1, not week 8 # COMMUNICATE uncertainty honestly — false precision destroys trust
✅ Do
- Decompose work, estimate in ranges, and state your confidence
- Identify and communicate risks/unknowns explicitly and early
- Prototype the riskiest/most-unknown part first to shrink the range
❌ Don't
- Give a single false-precise number — you'll break the promise and trust
- Hide risk to sound confident — surfaced risk is a gift, hidden risk is a landmine
22What a tech lead actually does★ core idea▶
Scenario: you're a great engineer, so you're made tech lead — and you keep trying to write all the code yourself while the team waits on you and the actual leadership work goes undone. The job changed; your impact now flows through others, not just your keyboard.
What
A tech lead multiplies the team's output: setting technical direction, unblocking people, ensuring quality, making decisions, and growing engineers — while writing less code than before.
Why
Your individual output caps out; a team's doesn't. The role's leverage is making the team more effective, not being the best individual contributor.
How
Shift from "how much can I build?" to "how much can I enable?" — direction, decisions (topic 25), reviews (topic 23), mentoring (topic 24), and communication (topic 29), plus some hands-on work.
IC (before): your impact = what YOU build (caps at one person)
TECH LEAD (now): your impact = what the TEAM builds (scales with people)
the multiplier work: direction · unblocking · decisions · reviews ·
mentoring · communication · a LITTLE hands-on code
trap: staying heads-down coding → team stalls waiting on you
# the shift in where your hours go: # direction — where are we going & why? (align the team, topic 29) # unblocking — remove obstacles so others move fast ← high leverage # decisions — make the calls others are stuck on (topic 25) # quality — reviews (topic 23), standards, testing culture # growth — mentoring, delegating stretch work (topic 24) # SOME code — stay technical & credible, but don't be the bottleneck # measure success by the TEAM's output & growth, not your commit count
✅ Do
- Optimize for the team's output — unblock, decide, direct, grow people
- Stay technical enough to be credible, but delegate the bulk of the coding
- Treat "the team is faster without waiting on me" as a success metric
❌ Don't
- Hoard the interesting work — you become a bottleneck and starve growth
- Measure yourself by your own commit count — that's the IC scoreboard
23Code review as leadership★ core idea▶
Scenario: a junior submits a PR with a flawed approach. You can rewrite it yourself (fast, but they learn nothing and feel small), or nitpick every style detail (slow, demoralizing), or use the review to teach and raise the bar. Code review is one of a lead's highest-leverage tools — for quality and for growing people.
What
Reviewing code not just to catch bugs but to teach, share context, maintain standards, and build a culture — with feedback that's kind, specific, and focused on what matters.
Why
Reviews are where standards propagate, engineers learn, and knowledge spreads. Done well they lift the whole team; done badly they demoralize, bottleneck, and teach nothing.
How
Focus on correctness, design, and clarity over nitpicks (automate style). Explain the why, ask questions, praise good work, and separate "must fix" from "consider."
automate the trivial: formatters & linters catch style → don't nitpick it
focus human review on: correctness · design · clarity · security
give feedback that is:
KIND (assume good intent) · SPECIFIC · explains the WHY
labeled: [blocking] must-fix vs [nit]/[optional] suggestion
a review is a teaching moment + a culture signal, not a gate to guard
# AUTOMATE style (formatter/linter) → humans review substance, not commas # PRIORITIZE feedback so the author knows what matters: # [blocking] this SQL is injectable — parameterize it ← must fix # [suggestion] extracting this into a helper would read cleaner # [nit] tiny naming preference (optional) # [praise] nice edge-case handling here 👏 ← call out good work! # EXPLAIN the why: "this can race under concurrent writes because..." # ASK, don't decree: "what happens if `items` is empty here?" # rewriting it yourself is fast but teaches nothing → guide, don't seize
✅ Do
- Automate style; focus human review on correctness, design, and clarity
- Be kind and specific, explain the why, and label blocking vs optional
- Praise good work and ask questions rather than only decreeing changes
❌ Don't
- Nitpick style a linter should catch — it's noise that buries real feedback
- Rewrite it yourself or block for days — you teach nothing and bottleneck the team
24Mentoring & growing engineers▶
Scenario: a junior is stuck, and you're tempted to just hand them the answer — it's faster. But do that every time and they never grow, and you're forever their crutch. Growing engineers is how you scale beyond yourself, and it's often a lead's most lasting impact.
What
Deliberately developing engineers: giving stretch work with support, coaching over telling, sharing context, and creating the safety to learn from mistakes.
Why
A team that grows compounds — mentored engineers become senior, then mentor others. It's the highest-leverage, longest-lasting thing a lead does, and it multiplies capacity.
How
Delegate meaningful work slightly beyond current level; coach with questions instead of answers; give timely, specific feedback; and let people struggle productively (with a safety net).
TELL (fast, no growth): "do it like this" → they depend on you forever
COACH (slower, compounds): "what have you tried? what might happen if...?"
→ they build the skill to solve the NEXT one themselves
growth = stretch work (just beyond current level) + support + safety to fail
the goal: make yourself progressively LESS needed
# DELEGATE stretch work: assignments slightly beyond current level # (with a safety net) — growth happens at the edge of ability # COACH with questions, don't just answer: # "what have you tried?" "what do you think will happen if...?" # "what are the tradeoffs you see?" → builds THEIR judgment # FEEDBACK: timely, specific, kind; balance growth areas with genuine praise # CREATE SAFETY: let them make (recoverable) mistakes & own the fix # your success metric: they need you LESS over time (and start mentoring others)
✅ Do
- Coach with questions and share context, rather than just handing over answers
- Delegate stretch work (just beyond current level) with a safety net
- Give timely, specific feedback and create safety to learn from mistakes
❌ Don't
- Always give the answer — it's faster once but creates permanent dependence
- Only delegate boring work — people grow on meaningful, challenging tasks
25Technical decision-making & driving alignment★ core idea▶
Scenario: the team's split three ways on an approach and going in circles. Someone has to drive to a decision — not by decree, but by building enough alignment that the team commits and moves. Turning debate into committed action is core leadership.
What
Making sound technical decisions and getting the team genuinely bought in — gathering input, weighing tradeoffs (topic 19), deciding, and building alignment so people commit even if they disagreed.
Why
Indecision stalls teams; decree without buy-in breeds resentment and half-hearted execution. Aligned decisions move fast and stick, because the team owns them.
How
Gather input widely, make the tradeoffs explicit, decide with a clear owner, and use "disagree and commit" — everyone's heard, one direction is chosen, all row together.
✗ endless debate → team stalls, momentum dies
✗ decree from on high → resentment, half-hearted execution
✓ GATHER input (people support what they helped shape)
→ make TRADEOFFS explicit (topic 19)
→ DECIDE with a clear owner & deadline
→ "DISAGREE AND COMMIT": heard, decided, everyone rows together
# 1) GATHER input — people support what they helped shape; you also # learn things you'd have missed. (but input != consensus required) # 2) FRAME the tradeoffs explicitly (topic 19) so the choice is legible # 3) DECIDE — name an owner and a deadline; don't let it drift # 4) ALIGN via "disagree and commit": # everyone is heard → a direction is chosen → EVERYONE commits & rows, # even those who preferred another option # 5) WRITE IT DOWN (ADR, topic 19) so the decision & why endure # note: a good-enough decision made now often beats a perfect one made late
✅ Do
- Gather input widely, make tradeoffs explicit, then decide with a clear owner
- Use "disagree and commit" so the team moves together after a decision
- Write decisions down (ADRs, topic 19) to prevent re-litigation
❌ Don't
- Let debate run forever seeking full consensus — indecision has a cost too
- Decree without gathering input — you'll get compliance, not commitment
26RFCs & design docs▶
Scenario: a big project is about to start and everyone has a different picture in their head of what's being built. Writing a design doc (or RFC) before coding surfaces disagreements, gathers feedback, and creates shared understanding — catching expensive mistakes when they're still cheap to fix (on paper).
What
A written proposal for a non-trivial change — problem, goals, proposed design, alternatives, tradeoffs, risks — circulated for feedback (RFC = Request For Comments) before building.
Why
Writing forces clarity and exposes gaps; review catches flaws when they're cheap (a comment, not a rewrite); and it aligns everyone and creates a durable record of the why.
How
Write context → goals/non-goals → proposed design → alternatives considered → risks/open questions. Share it, invite critique, iterate, then build with alignment.
RFC / design doc sections:
CONTEXT & PROBLEM what & why (the motivation)
GOALS / NON-GOALS scope — what we will & won't do
PROPOSED DESIGN the approach (diagrams welcome)
ALTERNATIVES what else we considered & why not (topic 19)
RISKS / OPEN Qs what could go wrong; unknowns
→ circulate → gather comments → iterate → THEN build (aligned)
catching a design flaw here costs a comment; catching it in prod costs weeks
# RFC / Design Doc — write it BEFORE building non-trivial things # # TITLE, author, date, status (draft / in-review / accepted) # CONTEXT & PROBLEM — what are we solving and why now? # GOALS / NON-GOALS — scope; explicitly what we're NOT doing # PROPOSED DESIGN — the approach; diagrams; data model; API changes # ALTERNATIVES — options considered & why rejected (topic 19) # RISKS & OPEN Qs — what could go wrong; unknowns; migration plan (18) # # then: circulate → collect comments → iterate → get sign-off → build # writing forces the clarity that vague head-plans hide
✅ Do
- Write a design doc for non-trivial work before building; circulate for feedback
- Include goals/non-goals, alternatives considered, and risks/open questions
- Use it to align the team and preserve the why (pairs with ADRs, topic 19)
❌ Don't
- Skip the doc and start coding a big project — misalignment surfaces expensively later
- Write a doc no one reviews — the value is in the feedback and shared understanding
27Leading migrations & large changes▶
Scenario: the org needs to move off a legacy system, or re-platform, or adopt a new architecture — a months-long change touching many teams. These large efforts fail not on the tech (topic 18 covers the how) but on leadership: sequencing, momentum, buy-in, and not letting it stall halfway.
What
The leadership side of large changes: breaking a big migration into shippable increments, sequencing them, keeping momentum and morale, securing buy-in, and driving it to done — not 80%-done forever.
Why
Big changes have no single "ship" moment — they're long, unglamorous, and easy to abandon halfway (leaving two systems to maintain). Leadership is what carries them across the line.
How
Slice into incremental, independently-valuable steps (topic 18); show progress; keep stakeholders bought in; celebrate milestones; and relentlessly drive the last mile — killing the old system.
the danger zone of big migrations:
start (exciting) → ▓▓▓▓░░░░ 50% → interest fades → STALLS at ~80%
→ now you maintain BOTH old & new forever = worst of both worlds ✗
leadership job:
slice into shippable increments (each delivers value)
show progress · keep buy-in · celebrate milestones
RELENTLESSLY drive the last mile → decommission the old ✓
# SLICE the migration into increments that each deliver value & can ship # (expand/migrate/contract per piece — topic 18) # SEQUENCE: do a thin end-to-end slice first to prove the approach # MOMENTUM: make progress VISIBLE (dashboard: % migrated); celebrate milestones # BUY-IN: keep stakeholders bought in — migrations compete with features # THE LAST MILE is the hard part: # an 80%-done migration = maintaining TWO systems = worst of both # → relentlessly drive to 100% and DECOMMISSION the old system # a migration isn't done when the new thing works — it's done when the OLD is GONE
✅ Do
- Slice into incremental, independently-valuable, shippable steps (topic 18)
- Make progress visible, keep stakeholders bought in, celebrate milestones
- Relentlessly drive the last mile and decommission the old system
❌ Don't
- Let a migration stall at "80% done" — that's maintaining two systems forever
- Treat "the new thing works" as done — it's done when the old thing is gone
28Managing technical debt▶
Scenario: the codebase is slowing everyone down — every feature takes longer than it should. But you can't stop for a six-month "rewrite," and you can't ignore it either. Managing technical debt deliberately — not zero, not infinite — is a balance a lead owns.
What
Technical debt is the accumulated cost of past shortcuts and aging design that slows future work. Managing it means paying it down strategically — not eliminating it, not ignoring it.
Why
Too much debt grinds delivery to a halt; obsessing over zero debt means never shipping. The lead's job is keeping it at a level where the team stays productive.
How
Make debt visible, prioritize paying down what actually hurts (high-traffic, high-change areas), pay it continuously (a % of each cycle), and frame it to the business in terms of velocity.
too MUCH debt: every feature is slow & risky → delivery grinds down
ZERO debt goal: you never ship (perfect is the enemy of done)
RIGHT: keep it manageable — pay down what HURTS, tolerate the rest
prioritize by PAIN: high-traffic + high-change code = pay first
rarely-touched ugly corner? leave it — debt only costs when you touch it
pay CONTINUOUSLY (a slice each cycle), not in a doomed big-bang rewrite
# MAKE IT VISIBLE: track debt like work (tickets, a register) — invisible # debt never gets prioritized against features # PRIORITIZE BY PAIN: debt only costs you when you TOUCH that code # high-traffic + frequently-changed & messy → pay down FIRST # ugly but stable & rarely-touched → usually leave it # PAY CONTINUOUSLY: budget ~10-20% of each cycle for debt/maintenance # (a steady diet beats a doomed 6-month 'stop everything and rewrite') # FRAME TO THE BUSINESS in velocity: "this refactor makes the next 5 # features 30% faster" — not "the code is ugly"
✅ Do
- Make debt visible and prioritize by pain (high-traffic, high-change code first)
- Pay it down continuously — a slice each cycle, not a big-bang rewrite
- Frame debt work to the business in terms of velocity and risk, not aesthetics
❌ Don't
- Chase zero debt or a giant "stop everything and rewrite" — both fail
- Refactor stable code no one touches — debt only costs when you work in it
29Communication & influence — up, down, across★ core idea▶
Scenario: you have the right technical answer, but you can't get the team to adopt it, can't get leadership to fund it, and can't get another team to cooperate. Technical skill without communication and influence hits a ceiling fast. At the senior level, how you communicate is the job.
What
Communicating and influencing effectively in every direction: up (leadership — outcomes, tradeoffs, risk), down (your team — context, clarity), and across (peer teams — alignment without authority).
Why
Ideas don't implement themselves — they need buy-in. A senior engineer's leverage comes from influence, and most of that is tailoring the message to the audience.
How
Speak each audience's language: business impact for leadership, technical context for the team, mutual benefit for peers. Listen first, write clearly, and lead without relying on authority.
UP (leadership): outcomes, cost, risk, tradeoffs — the BUSINESS impact
(not implementation details they don't need)
DOWN (your team): context & the WHY, clear priorities, room to own it
ACROSS (peers): shared goals, mutual benefit — influence w/o authority
same idea, three translations. Influence = making OTHERS want it.
listen first · write clearly · assume good intent
# UP — to leadership: lead with the OUTCOME and the TRADEOFF # ✗ "we should refactor the auth service's token module" # ✓ "a 2-week investment cuts login failures 40% & unblocks SSO — the # tradeoff is delaying feature X by 2 weeks. Recommend we do it." # DOWN — to your team: give CONTEXT (the why), clear priorities, ownership # people execute better on decisions they understand & own (topic 25) # ACROSS — to peer teams (no authority): find MUTUAL benefit # "this helps your latency too" beats "please do this for me" # ALWAYS: listen first · write clearly · assume good intent · be concise
✅ Do
- Tailor the message: business impact up, context down, mutual benefit across
- Listen first, write clearly and concisely, and assume good intent
- Build influence through relationships and shared goals, not authority
❌ Don't
- Bury leadership in implementation detail or your team in vague vision
- Assume being right is enough — unadopted good ideas have zero impact
30Capstone — leading a system from proposal to production★ capstone▶
Scenario: put it all together — you're handed an ambiguous, high-stakes project ("build the new notifications platform") and must take it from a blank page to running in production, leading both the system and the people. This is the whole playbook working together.
What
The full arc: clarify → design (with tradeoffs) → write it up (RFC) → align the team → build incrementally with reliability & security baked in → migrate safely → operate against SLOs — while growing the people who build it.
Why
Any one skill is learnable alone. Senior leadership is making the technical and the human work together: the right system, built by a growing team, shipped safely, and operated well.
How
Requirements & estimation → design + ADR/RFC → drive alignment → slice into increments → build with observability, resilience, security → migrate expand/contract → run to SLOs → retro & grow the team.
CLARIFY + ESTIMATE (1,2) → DESIGN w/ tradeoffs (19) → RFC + align (26,25)
│
▼ build incrementally, with NFRs baked in:
scaling (3,7) · caching (5) · async (9) · resilience (15)
observability (14) · security (17) · SLOs + error budget (13)
│
▼ migrate safely (18) → operate → retro
THROUGHOUT (the human half): review (23) · mentor (24) · decide (25)
communicate up/down/across (29)
done = shipped, reliable, the old system gone (27), team leveled up
# 1. CLARIFY requirements + NFRs (1,12); ESTIMATE scale (2) # 2. DESIGN with explicit tradeoffs (19); write an RFC (26); ALIGN team (25) # 3. SLICE into shippable increments; sequence a thin end-to-end slice first # 4. BUILD with the non-functionals baked in from day one: # scale (3,7) · cache (5) · async where it fits (9) # resilience/blast-radius (15) · observability (14) · security (17) # define SLIs/SLOs + error budget (13) # 5. MIGRATE safely — expand/migrate/contract, reversible (18) # 6. OPERATE to SLOs; run a blameless retro; pay down debt continuously (28) # THE HUMAN HALF, throughout: review to teach (23), mentor & delegate (24), # decide & drive alignment (25), communicate up/down/across (29) # DONE = shipped + reliable + OLD system gone (27) + the team leveled up
✅ Do
- Lead the system and the people — the technical and human halves are one job
- Bake in reliability, security, and observability from day one, not after the incident
- Ship incrementally, migrate reversibly, and grow the team as you build
❌ Don't
- Optimize only the architecture and neglect alignment, growth, and communication
- Call it done when the new system works — it's done when it's reliable, adopted, and the old one is gone
Answer 5 per round. Your progress and score are saved in this browser — come back anytime and continue where you left off.