Case Study · Scaling · Distributed Systems

Scaling Chirp to 1,000,000 req/s

Chirp is a social feed that went from 10k users on one box to 40M users at a million requests a second. This is the ladder it climbed — and the one idea that explains every rung: the bottleneck moves, and it always ends up at the database.

A worked application of the Distributed Systems & System Design playbooks

0The wall

Chirp launched on one server: an app process and a PostgreSQL database on the same box. It was fine at 10k users. At 40M users and peak traffic, that design would need to serve ~1,000,000 requests/sec — and it fell over long before. The question isn't "how do we make the server faster" — it's "where is the bottleneck, and what do we do when it moves?"

🧠 The one idea behind every rung of the ladder

Every layer of a system is either stateless or stateful, and they scale in opposite ways:

  • Stateless layers (CDN, load balancer, app servers) remember nothing between requests, so any copy can serve anyone. You scale them by cloning — add identical boxes behind a load balancer. This is easy, and it's exactly what Kubernetes and autoscaling automate.
  • Stateful layers (the database) hold the shared truth, so you can't just run 50 independent copies that all disagree. This is the hard part.

So a flood of requests fans out cheaply across the cloneable layers and then funnels onto the one thing you can't trivially clone: Postgres. That's why nearly every scaling technique below — replicas, cache, sharding, queues — is really about protecting or splitting the stateful layer. Scale the easy stuff first; then spend your real effort on the database.

1Read the traffic before you scale it

You can't scale what you haven't measured. Chirp's traffic is overwhelmingly reads (people scroll far more than they post), and it's fan-out heavy (one celebrity post reaches millions of timelines). That shape dictates the whole strategy: optimize reads hard, make writes asynchronous.

RequestShare of trafficRead/WriteLatency budgetScaling lever
Load timeline~80%read< 100mscache + read replicas
Post a chirp~5%write< 300ms (async fan-out)queue + outbox
Notifications / counts~10%read< 200mscache, precomputed
Search / profile~5%read< 300msdedicated index/store
Insight 1
Reads dominate
80%+ of load is reading timelines.
→ caching + read replicas win the most.
Insight 2
Writes fan out
One post → millions of timelines.
→ do the fan-out async, off the request path.
Insight 3
Not all data is equal
A stale like-count is fine; a failed post is not.
→ spend consistency budget only where it matters.

2The architecture, three ways

Here's the shape Chirp grew into — the same system as an ASCII request path, an entity diagram of the sharded data model, and a sequence diagram of one timeline read.

Request path — fan out cheap, funnel onto state
   1,000,000 req/s
        │
        ▼   CDN / edge cache ........ static + hot timelines (absorbs a huge %)
        ▼   Load balancer ........... spreads what's left
        ▼   [ app ][ app ][ app ]×N . STATELESS — just add clones (autoscale)
        ▼   Redis cache ............. timelines & counts (most reads stop here)
        │         │ miss
        ▼         ▼
   write path   read replicas ×M .... scale READS by cloning the data
   (queue →     │
    async       ▼
    fan-out)   PRIMARY (sharded) .... the stateful WALL — split by user_id
               shard0 shard1 shard2 ... shardK
   easy to clone ↑            ↑ the hard part: one source of truth per shard
Entity diagram — the sharded data model
erDiagram USER ||--o{ CHIRP : posts USER ||--o{ FOLLOW : "follows (edge)" USER ||--|| TIMELINE : has TIMELINE ||--o{ TIMELINE_ENTRY : contains CHIRP ||--o{ TIMELINE_ENTRY : "fanned into" SHARD ||--o{ USER : "owns by user_id" USER { bigint user_id "shard key" string handle } CHIRP { bigint chirp_id bigint author_id "shard key" text body datetime created_at } FOLLOW { bigint follower_id "shard key" bigint followee_id } TIMELINE_ENTRY { bigint owner_id "shard key" bigint chirp_id datetime created_at } SHARD { int shard_id string range "user_id range" }
Renders with Mermaid. The ASCII request path above carries the same structure.
Sequence diagram — one timeline read (cache hit vs miss)
sequenceDiagram autonumber participant C as Client participant E as CDN / edge participant LB as Load balancer participant App as App server participant R as Redis cache participant DB as Read replica (shard) C->>E: GET /timeline E-->>C: hot timeline (edge hit) — many requests end here C->>LB: GET /timeline (edge miss) LB->>App: route to any stateless app clone App->>R: GET timeline:user alt cache hit (~90%) R-->>App: cached timeline else cache miss R-->>App: miss App->>DB: read recent entries for user (from their shard) DB-->>App: rows App->>R: SET timeline:user (TTL + jitter) end App-->>C: timeline JSON Note over App,DB: reads scale by CLONING (replicas + cache); writes fan out async
Renders with Mermaid. The steps are described in Section 3.

3The scaling ladder

Chirp climbed one rung at a time, and only when a real bottleneck forced it — because every rung adds complexity you'll debug at 3am. Each stage below is a symptom, the move, and the trap. Expand them.

1Scale up — one bigger boxbuys time, cheaply▶

Symptom: CPU/RAM maxed on the single server; latency creeping up.

The first and most underrated move: give the box more CPU, RAM, and faster disk. Vertical scaling has no distributed-systems complexity — no consistency, no coordination — and it buys real headroom fast. Do this before anything clever.

Trap: vertical scaling has a ceiling and a single point of failure — one box is still one box. It's a bridge to the next rung, not a destination. Don't over-invest in one giant server when you'll need to split anyway.
2Split the app tier — stateless clones behind a load balancer▶

Symptom: the app process is the bottleneck (CPU-bound), or you need zero-downtime deploys and failover.

Move the database off the app box, then run many identical app servers behind a load balancer. Because the app is stateless (no session state on the box — put sessions in Redis/tokens), any clone serves any request, and autoscaling adds/removes them with load. This is the cheap, near-infinite part of scaling.

✅
The easy winThis rung absorbs enormous traffic for little conceptual cost — it's just cloning. The moment your app holds no per-box state, you can add app servers all day. The flood now funnels one layer down, onto the database.
Trap: hidden state kills statelessness — a local session, an in-memory cache that must be consistent, files on local disk. Externalize all of it (Redis, S3, tokens) or your "stateless" clones quietly disagree.
3Read replicas — scale reads by cloning the data▶

Symptom: the database is read-saturated; 80% of load is timelines and the primary can't keep up.

Add read replicas — copies of the database that stream changes from the primary and serve read-only queries. Route reads to replicas, writes to the primary. Since Chirp is read-heavy, this multiplies read capacity by adding replicas — cloning, applied to (mostly-read) data.

Trap: replication lag breaks "read-your-writes" — a user posts, then their next read hits a replica that hasn't caught up, and their post is missing. Route a user's reads to the primary (or a synced replica) right after they write, and watch lag as a first-class metric.
4Cache — stop most reads before they reach the DB▶

Symptom: even with replicas, the same hot timelines are read millions of times; the DB does redundant work.

Put Redis in front of the database. Cache assembled timelines, counts, and hot objects with a TTL. At a ~90% hit rate, nine of ten reads never touch Postgres — the cheapest scaling you can buy. Combined with the edge/CDN, most reads end far from the stateful layer.

Trap: the hardest problem in computer science — invalidation — plus the thundering herd: a hot key expires and thousands of requests stampede the DB at once to rebuild it. Add TTL jitter and a rebuild lock so exactly one request refills a cold key.
5Shard — split the stateful layer itselfthe hard rung▶

Symptom: writes and total data outgrow a single primary; one Postgres can't hold or write it all, even with replicas (replicas scale reads, not writes/size).

Split the data across shards — independent databases, each owning a slice of users by user_id. Now writes and storage scale horizontally: shard 3 knows nothing about shard 7. This is the real answer to a stateful bottleneck — you stop cloning the whole thing and start partitioning it.

Trap: the shard key is a near-permanent decision, and cross-shard queries (a search across all users, a join spanning shards) get painful. Pick a key that matches your access pattern (Chirp reads a user's own timeline → shard by user_id), and resharding later is a major migration — plan capacity ahead.
6Go async — queues, fan-out, and the outbox▶

Symptom: a single post must write to millions of follower timelines; doing that synchronously makes posting slow and fragile.

Take heavy work off the request path. A post writes the chirp and drops a job on a queue (Kafka/SQS); background workers fan it out to follower timelines. The user's post returns in milliseconds; the fan-out happens behind the scenes. Use the outbox pattern so the write and the event are atomic, and design consumers to be idempotent (the same fan-out job may run twice).

Trap: the celebrity / hot-key problem — fanning a post from an account with 50M followers into 50M timelines is a storm. Hybrid it: pre-fan-out for normal users, but pull celebrity posts at read time (merge them into the timeline on the fly) so one post doesn't write 50M rows.

4What breaks at a million per second

The rungs get you the throughput; these are the failure modes that show up because you scaled, and the teams that survive them planned for them.

The four that page you at scale
  THUNDERING HERD    a hot cache key expires → thousands rebuild it at once
                     fix: TTL jitter + single-flight rebuild lock
  HOT SHARD          one celebrity / one popular key overloads a single shard
                     fix: hybrid fan-out (push normal, pull celebrity); split hot keys
  REPLICATION LAG    replica behind → stale reads, broken "read-your-writes"
                     fix: read from primary after a write; alarm on lag vs budget
  RETRY STORM        a blip → everyone retries → self-inflicted DDoS
                     fix: backoff + jitter, circuit breakers, load shedding (429)
🧯
The meta-lessonEach fix here is straight from the Distributed Systems playbook — idempotency, backpressure, circuit breakers, tunable consistency. Scaling doesn't remove those concerns; it's what creates them. The throughput is the easy half; staying up while you have it is the craft.

5Scorecard — climb a rung only when it's forced

The senior skill isn't knowing these techniques — it's knowing when not to reach for them yet. Each rung is complexity you'll debug forever. Add one only when the box below it is genuinely the bottleneck.

1
Scale up first — a bigger box has no distributed complexity and buys real time.
2
Stateless app tier + LB when the app is the bottleneck or you need HA — cheap, near-infinite.
3
Read replicas when reads saturate the DB — and only after you've handled replication lag.
4
Cache when the same reads repeat — the highest-leverage rung; mind invalidation + the herd.
5
Shard only when writes/size outgrow one primary — the shard key is forever; model access first.
6
Async / queues when heavy work belongs off the request path — idempotent consumers, outbox, hybrid fan-out.
The one-line takeaway: stateless layers scale by cloning (easy); the stateful database is the wall, and every rung — replicas, cache, shards, queues — is a different way to protect or split it. Measure first, climb only when forced, and remember that the throughput is free compared to the resilience you'll need to keep it up.