Networking Pro Playbook — 26 Topics on the Wire

Part I — Networking · The Wire

How Packets Travel

01The layer model — the postal system▶

Scenario: "is it a DNS problem, a routing problem, a firewall problem, or an app problem?" The layer model is the map that tells you where to look — and the order to debug in (topic 23).

🧠

Analogy: sending a package through the postal system. You write a letter (the app data — HTTP), seal it in an envelope with a full address (TCP/IP), the post office routes it city-to-city (IP routing), trucks carry it road-by-road (Ethernet/wifi — the physical hop). Each layer only cares about its own job and trusts the layer below. A failure at any layer looks different — and you debug bottom-up.

The layers, mapped to what you actually touch

Layer            Deals with          You debug it with
─────────────────────────────────────────────────────────
App (HTTP/TLS)   the actual request  curl, browser devtools (topics 11,12,17)
Transport (TCP)  ports, reliability  ss, telnet, nc      (topics 5,7)
Network (IP)     addresses, routing  ping, traceroute, ip route (topics 2,4,15,16)
Link (Ethernet)  the local hop, MAC  arp, ip link        (topic 3)

# the two models: OSI has 7 layers (academic), TCP/IP has 4 (practical).
# You mostly reason in 4: App, Transport, Internet, Link.
# Mnemonic for encapsulation: each layer WRAPS the one above in its own header.

One HTTP request, unwrapped (what tcpdump shows you — topic 52)

[ Ethernet | IP | TCP | TLS | HTTP GET /api ]
  └MAC→MAC  └IP→IP └port └encrypt └the actual request
  each layer's header wraps the payload of the layer above — like nested envelopes.

✅ Do — good use cases

Debug bottom-up: is the host reachable (IP/ping)? port open (TCP/ss)? DNS resolving? app responding? (topic 23 is this as a runbook)
Map each symptom to a layer — "connection refused" is transport, "name not resolved" is DNS, "403" is app
Reason in the 4-layer TCP/IP model; leave OSI's 7 layers for exams

❌ Don't — anti-patterns

Debugging top-down ("must be the app") when the host isn't even pingable — you'll waste an hour above a broken lower layer
Conflating layers — a working ping (IP) says nothing about whether the port (TCP) or app is up

Pattern & benefit: the layer model is the master index for every networking tool and problem — each tool operates at one layer, each error belongs to one layer, and debugging in layer order (bottom-up) finds the break fast instead of guessing. Every remaining topic in this part slots into a layer.

Gotcha — why it goes wrong: the classic time-sink is debugging the wrong layer — spending an hour on app config when ping already fails (the box is unreachable at the IP layer), or blaming the network when it's a DNS (name→IP) issue one layer up from routing. The discipline "confirm each layer before climbing to the next" (topic 23) is what separates a 5-minute diagnosis from an afternoon.

02IP addresses & CIDR — the addressing system★ used daily▶

Scenario: "is 10.0.5.20 in the 10.0.0.0/16 subnet? can these two hosts talk directly? what does /24 mean?" CIDR is the math behind every VPC, firewall rule, and route.

🧠

Analogy: an IP address is a street address; the subnet (the /N) is the postcode that says which houses share a street. A /24 is a street of 256 houses (10.0.5.0–10.0.5.255); a /16 is a whole town of 65,536. Houses on the same street talk directly (local delivery); different streets go through the post office (the gateway/router — topic 38). The /N is literally how many address bits are "the street," leaving the rest for "the house."

Code

# CIDR: /N = how many bits are the NETWORK; the rest are HOSTS.
10.0.5.0/24    → network 10.0.5.x, hosts .1–.254   (256 addresses, ~254 usable)
10.0.0.0/16    → 10.0.x.x                          (65,536 addresses)
0.0.0.0/0      → "everything" (default route / allow-all firewall rule)

# private ranges (RFC1918) — free to use internally, never routed on the internet:
10.0.0.0/8      172.16.0.0/12      192.168.0.0/16

$ ipcalc 10.0.5.20/24        # do the math for you
Network:   10.0.5.0/24
HostMin:   10.0.5.1   HostMax: 10.0.5.254   Broadcast: 10.0.5.255
$ ip -c addr                 # this host's addresses + their CIDR
inet 10.0.5.20/24 ...        ← "I'm .20 on the 10.0.5.0/24 street"

The everyday question: same subnet = direct talk?

10.0.5.20/24  and  10.0.5.99   → same /24 street → talk DIRECTLY (topic 3)
10.0.5.20/24  and  10.0.9.99   → different streets → via the GATEWAY (topic 4)
smaller /N (e.g. /16) = bigger network;  bigger /N (e.g. /28) = smaller

✅ Do — good use cases

Read /N as "network size": /24=256, /16=65k, /8=16M — smaller number, bigger network
Use ipcalc/sipcalc for range math instead of doing binary in your head
Size VPC subnets generously — pod-per-IP designs (DevOps playbook EKS topic 39) drink addresses fast

❌ Don't — anti-patterns

Overlapping CIDR ranges across VPCs you'll ever peer/VPN — you can't route between two 10.0.0.0/16s
Reading /24 as "bigger than /16" — it's backwards; more prefix bits = fewer hosts

Pattern & benefit: CIDR is the one piece of networking math you truly must own — it defines who's local vs remote (topic 3 vs 4), what a firewall rule covers (topic 20), and how cloud VPCs carve up address space (topic 25). Once "same subnet = direct, else via gateway" clicks, routing stops being mysterious.

Gotcha — why it goes wrong: the prefix length is inverse to size and it trips everyone — /28 is a tiny 16-address block, /8 is enormous. Sizing a subnet /28 "to be safe" then running out of IPs is a classic (DevOps playbook EKS topic 34's exact warning). And two networks with overlapping ranges can never be joined — pick non-overlapping CIDRs from day one, because re-IP'ing later is a migration.

03MAC & ARP — the local delivery layer▶

Scenario: two machines on the same subnet (topic 2) actually reach each other by MAC address, not IP — and ARP is the "who has this IP?" broadcast that bridges the two. Rarely touched, but it explains a whole class of LAN weirdness.

🧠

Analogy: the IP address is someone's name; the MAC address is their actual face in the room. To hand a note to "Priya" on your street, you shout "who is Priya?" (ARP request), she says "me, here!" (ARP reply with her MAC/face), and now you deliver directly. Your address book (ARP cache) remembers her face for a while so you don't shout every time.

Code

$ ip neigh                    # the ARP cache: IP → MAC on the local link
10.0.5.1   dev eth0 lladdr 0a:1b:2c:3d:4e:5f REACHABLE   ← the gateway's MAC
10.0.5.99  dev eth0 lladdr 0a:aa:bb:cc:dd:ee STALE

$ ip link                     # this host's interfaces + THEIR MACs
2: eth0: ... link/ether 0a:11:22:33:44:55

# the sequence for talking to a same-subnet host:
#   1. IP layer: "10.0.5.99 is on my /24 → it's LOCAL, deliver directly"
#   2. ARP: broadcast "who has 10.0.5.99?" → reply "me, MAC 0a:aa:.."
#   3. send the frame to that MAC. (Different subnet? ARP for the GATEWAY instead.)
$ arping 10.0.5.99            # probe a specific IP at layer 2

Output — a duplicate-IP incident, seen in ARP

$ ip neigh | grep 10.0.5.50
10.0.5.50 dev eth0 lladdr 0a:aa:.. REACHABLE
# ...intermittent connectivity to .50? two machines answering ARP for one IP
# = IP conflict. ARP is where you catch it.

✅ Do — good use cases

ip neigh to inspect the ARP cache when same-subnet connectivity is flaky
Suspect an IP conflict (two hosts, one IP) when a LAN address is intermittently reachable — ARP shows the flapping MAC
Understand that same-subnet = MAC delivery, cross-subnet = deliver to the gateway's MAC (topic 4)

❌ Don't — anti-patterns

Reaching for ARP on cross-subnet problems — ARP is local-link only; remote reachability is routing (topic 4)
Ignoring duplicate-IP symptoms — "works, then doesn't, then works" on a LAN is often two devices fighting over one address

Pattern & benefit: ARP is the invisible glue between the IP world (topic 2) and the physical wire — knowing it exists explains why same-subnet hosts talk "directly," why cross-subnet traffic goes to the gateway's MAC, and how to catch IP conflicts. You'll rarely touch it, but when LAN connectivity is haunted, ip neigh is where the ghost shows up.

Gotcha — why it goes wrong: a stale or poisoned ARP cache causes "the IP is right but traffic goes nowhere" — the host is delivering frames to a MAC that moved or never existed. Clearing the entry (ip neigh flush) or waiting for it to expire fixes it. And in clouds, ARP is largely virtualized away (the SDN handles it), which is why you rarely debug it on EC2 — but on bare metal and home/office LANs it's very real.

04Routing & gateways — how packets leave the subnet★ used daily▶

Scenario: a packet for 142.250.x.x (Google) isn't on your subnet — where does it go? The routing table decides the next hop for every destination, and the default route is the catch-all to the outside world.

🧠

Analogy: the routing table is a signpost at every intersection. For each destination it says "that way." Local street? Deliver directly (topic 3). Anywhere else? "Take it to the post office" — the default gateway (0.0.0.0/0), the road out of town. The most specific sign wins: a sign for "123 Main St" overrides the general "everything else → gateway."

Code

$ ip route                    # the routing table (signposts)
default via 10.0.5.1 dev eth0        ← catch-all: everything → the gateway
10.0.5.0/24 dev eth0 proto kernel    ← my local subnet: deliver directly
$ ip route get 142.250.72.14  # ask "how would I reach THIS?"
142.250.72.14 via 10.0.5.1 dev eth0   ← via the gateway (it's not local)
$ ip route get 10.0.5.99
10.0.5.99 dev eth0 src 10.0.5.20      ← direct (same subnet, topic 37)

# the decision, every packet:
#   dest in a local subnet route? → deliver directly (ARP for it)
#   else → most-specific matching route's next-hop
#   else → default route (the gateway)
#   no default + not local? → "Network is unreachable"

Output — "no route" vs "reached the gateway"

$ ip route get 8.8.8.8            # healthy:
8.8.8.8 via 10.0.5.1 dev eth0    ← knows the way out
# broken (no default route):
$ ping 8.8.8.8
connect: Network is unreachable   ← no signpost for "everything else"

✅ Do — good use cases

ip route get <dest> to see exactly how THIS host would reach a destination — the fastest routing debug
Confirm a working default route (default via …) when a host can reach its subnet but not the internet
Read "most specific route wins" — a /32 host route overrides the /0 default

❌ Don't — anti-patterns

Assuming "can ping the gateway but not the internet" is DNS — check the gateway's own upstream/NAT (topic 8) and your default route first
Adding overlapping/conflicting static routes by hand without understanding specificity — you can silently blackhole traffic

Pattern & benefit: routing is the layer that turns "my subnet" into "the whole internet" — and ip route get makes it inspectable per-destination, so "why can't this host reach X" becomes a one-command answer. It's the same model in cloud VPC route tables (topic 25): local routes plus a default route to an internet/NAT gateway.

Gotcha — why it goes wrong: "reaches its own subnet but nothing beyond" is almost always a missing or wrong default route — Network is unreachable is the kernel saying "no signpost for that destination." And a subtle one: you can ping the gateway (it's on your subnet) yet have no internet if the gateway's upstream/NAT is broken — reaching the gateway proves the local hop, not the path beyond it (debug in layers, topic 57).

05TCP — the reliable handshake★ used daily▶

Scenario: nearly everything you deploy (HTTP, SSH, databases) rides on TCP. Its three-way handshake and connection states explain "connection refused" vs "timed out" — two very different failures.

🧠

Analogy: TCP is a phone call with confirmations. The handshake is "Can you hear me?" → "Yes, can you hear me?" → "Yes" (SYN → SYN-ACK → ACK) before anyone talks. Then every sentence gets an "mm-hm" (ACK); miss one and it's repeated (retransmission). Compare to UDP (topic 6), which is shouting a postcard and hoping. TCP guarantees ordered, complete, acknowledged delivery — at the cost of that setup chatter.

Code — the handshake & what failures mean

# the 3-way handshake (what tcpdump shows — topic 52):
#   client → SYN         "let's talk, my seq=X"
#   server → SYN-ACK     "ok, my seq=Y, ack X+1"
#   client → ACK         "got it" → connection ESTABLISHED

# test if a TCP port is open (no app data, just the handshake):
$ nc -zv db.prod 5432
Connection to db.prod 5432 port [tcp] succeeded!     ← handshake completed
$ nc -zv db.prod 5432
nc: connect to db.prod port 5432 (tcp) failed: Connection refused
#                                              └ host UP, nothing LISTENING there
$ nc -zv db.prod 5432
nc: connect ... Operation timed out
#              └ no reply at all: firewall dropping, or host down (topic 20,23)

The two failures — memorize this distinction

Connection REFUSED  → reached the host; no service on that port (or it's down)
                      (the host actively said "no" — RST packet)
Connection TIMED OUT → no response at all; firewall DROP, wrong IP, or host dead
                      (silence — nobody answered the SYN)

✅ Do — good use cases

nc -zv host port (or /dev/tcp) to test reachability at the TCP layer — isolates network from app
Read "refused" as "host reachable, port closed" and "timed out" as "something is dropping/silent" — different fixes
Understand handshake cost — for many tiny requests, connection reuse/keep-alive matters (topic 11)

❌ Don't — anti-patterns

Treating "refused" and "timed out" as the same "can't connect" — one is a closed port, the other is usually a firewall (topic 20)
Blaming the app when the TCP handshake itself fails — the app never even saw the connection

Pattern & benefit: TCP's guarantees (ordered, complete, acknowledged) are why you can build reliable protocols on top without reinventing retries — and its failure modes are diagnostic gold: refused vs timed out instantly narrows a connectivity problem to "port/service" vs "firewall/routing" (topic 23 builds a runbook on exactly this).

Gotcha — why it goes wrong: the refused-vs-timeout distinction is the single most useful networking tell, and people conflate them. Refused = a RST packet came back: the host is up and reachable, but nothing is listening (service down, wrong port). Timed out = pure silence: a firewall is DROPping (not REJECTing) your SYN, the IP is wrong, or the host is dead. Same "can't connect" symptom, completely different root causes and fixes.

06UDP — fire and forget▶

Scenario: DNS (topic 9), video calls, game servers, metrics (statsd) — all UDP. No handshake, no guarantees, no retransmit. Knowing when that trade-off is right (and its debugging quirks) matters.

🧠

Analogy: if TCP is a phone call with confirmations, UDP is shouting a postcard over the fence: no "can you hear me?", no waiting for a reply, no resend if it's lost. Fast and cheap, but you never know if it arrived. Perfect when a late packet is worse than a lost one — in a video call, a re-sent frame from two seconds ago is useless; just move on.

Code

# UDP: no connection, no handshake, no delivery guarantee.
#   application handles what it needs (DNS retries, video conceals loss).

$ dig google.com              # DNS — classic UDP (port 53), falls back to TCP if big
$ nc -u -zv 10.0.5.1 514      # -u = UDP; "succeeded" here is WEAKER than TCP's:
#   UDP has no handshake, so "open" often just means "not actively refused"

# why UDP for these:
#   DNS      one tiny question, one answer; a retry is cheaper than a handshake
#   Video/VoIP  a lost frame is skipped; a LATE frame is useless
#   Metrics/statsd  fire thousands/sec; losing a few data points is fine
#   QUIC/HTTP3  builds its OWN reliability on UDP, dodging TCP's limitations

TCP vs UDP — the trade in one table

                TCP                        UDP
handshake       yes (setup cost)           none (instant)
delivery        guaranteed, ordered        best-effort, may drop/reorder
use when        correctness matters         speed/latency matters, loss tolerable
examples        HTTP, SSH, databases        DNS, video, games, metrics, QUIC

✅ Do — good use cases

UDP when low latency beats guaranteed delivery and the app tolerates/handles loss (media, metrics, DNS)
Remember DNS is UDP-first — a DNS issue can be a UDP/port-53 issue (firewalls sometimes block it, topic 44)
Expect app-level reliability on top of UDP (QUIC, DNS retries) rather than the transport providing it

❌ Don't — anti-patterns

Trusting nc -u "open" like a TCP success — with no handshake, UDP reachability tests are inherently ambiguous
Choosing UDP for data that must arrive intact (files, transactions) — that's TCP's job; don't rebuild TCP badly

Pattern & benefit: the TCP/UDP choice is the fundamental transport trade-off — reliability vs latency — and recognizing which a protocol uses tells you how it fails and how to debug it. DNS being UDP explains its speed and its occasional firewall/MTU quirks; QUIC/HTTP3 on UDP explains the industry's move to app-controlled reliability.

Gotcha — why it goes wrong: UDP reachability is genuinely hard to test — no handshake means nc -u often reports "succeeded" even when nothing's listening (the OS didn't get a rejection), so a "working" UDP test can lie. Confirm at the app layer (dig @server for DNS) instead. And large UDP packets can silently fragment or hit MTU limits (topic 24) — a DNS response too big for UDP triggers a TCP retry, which a port-53-TCP firewall block then breaks mysteriously.

07Ports & sockets — ss, the netstat successor★ used daily▶

Scenario: "what's listening on port 8080? is anything even bound? who's holding all these connections?" ss answers all of it — the first command in most connectivity debugging.

🧠

Analogy: a host is an apartment building; ports are numbered mail slots (0–65535). A service "listening" on a port is a tenant who put their name on slot 8080 and answers mail there. ss is reading the building directory: which slots have a tenant (LISTEN), which have active correspondence going (ESTABLISHED), and who they're talking to.

Code

# the flags to memorize: -t tcp  -u udp  -l listening  -n numeric  -p process
$ ss -tlnp                    # all TCP listeners + which process owns them
State   Local Address:Port    Process
LISTEN  0.0.0.0:80            users:(("nginx",pid=712))    ← world-facing
LISTEN  127.0.0.1:5432        users:(("postgres",pid=901)) ← localhost ONLY (safe)
LISTEN  *:22                  users:(("sshd",pid=688))

$ ss -tnp state established    # active connections + peers
$ ss -tn dst 10.0.5.99         # connections to a specific host
$ ss -s                        # summary counts by state (topic 19)

# "port already in use" — find the culprit:
$ ss -tlnp | grep :8080
$ sudo lsof -i :8080           # alternative, also shows the PID

Output — the security tell in one glance

LISTEN 0.0.0.0:5432   ← DANGER: postgres exposed to the WORLD
LISTEN 127.0.0.1:5432 ← SAFE: only reachable from this host
# 0.0.0.0 = "all interfaces, anyone can reach"; 127.0.0.1 = "this machine only"

✅ Do — good use cases

ss -tlnp as your first move for "is the service even up and bound?" — before blaming the network
Check the bind address: 0.0.0.0 = exposed to all, 127.0.0.1 = localhost-only (a security-critical distinction)
ss -tlnp | grep :PORT to resolve "address already in use" to a PID

❌ Don't — anti-patterns

Debugging "connection refused" (topic 5) from the client before checking ss on the server — the service may not be listening at all
Binding databases/admin ports to 0.0.0.0 without a firewall — you've published them to anyone who can route to the host

Pattern & benefit: ss is the bridge between "the app" and "the network" — it proves whether a service is actually listening, on which interface, and who's connected, turning vague "can't connect" reports into "postgres is bound to localhost only, that's why the remote client is refused." It replaces the older netstat and is faster on busy hosts.

Gotcha — why it goes wrong: the bind-address gotcha causes both bugs and breaches — a service on 127.0.0.1 works locally but refuses every remote client ("connection refused" from another host, fine from localhost), while a database accidentally on 0.0.0.0 is exposed to the entire network. The Local Address column is the answer to a huge fraction of "why can/can't X reach this service" questions.

08NAT — one public IP, many private hosts▶

Scenario: your laptop, your phone, and 50 servers in a VPC all share one public IP to reach the internet. NAT is the translation that makes private addresses (topic 2) work with the public internet — and it explains a lot of "why can't they reach in?" confusion.

🧠

Analogy: NAT is a company receptionist with one public phone number. Employees (private hosts) call out through her; she notes "extension 205 is calling Acme" and puts her number on the caller ID. Acme replies to the public number, she checks her notes, and routes it back to extension 205. Outbound works seamlessly — but Acme can't initiate a call to extension 205, because from outside, only the receptionist's number exists.

Code

# SNAT/masquerade: private source IP → public IP on the way OUT
#   10.0.5.20:44321 → [NAT] → 203.0.113.5:61000 → internet
#   reply comes back to 203.0.113.5:61000 → [NAT translates back] → 10.0.5.20:44321

$ curl ifconfig.me           # what the internet SEES as your source IP
203.0.113.5                  # not your private 10.0.5.20 — NAT rewrote it

# see NAT rules on a Linux router/host:
$ sudo iptables -t nat -L -n -v         # topic 54
# the cloud equivalents:
#   NAT Gateway  → private subnets reach OUT to the internet (egress)
#   but INBOUND needs a Load Balancer / public IP / port-forward (DNAT)

# port forwarding (DNAT) — the manual "let outsiders reach extension 205":
#   public :443 → private 10.0.5.20:8443

Output — the asymmetry that confuses everyone

OUTBOUND (private → internet):  works automatically via NAT/masquerade
INBOUND  (internet → private):  BLOCKED unless explicitly forwarded (DNAT)
                                 or fronted by a load balancer / public IP.
# "my server can pull updates but nobody can reach my app" = this, exactly.

✅ Do — good use cases

Understand NAT as directional: outbound is automatic, inbound needs explicit forwarding or a load balancer (topic 13)
curl ifconfig.me to see your public/NAT'd source IP — useful for allowlists and "which IP do I appear as?"
In cloud: NAT Gateway for private-subnet egress, LB/public IP for ingress — two different jobs

❌ Don't — anti-patterns

Expecting inbound connections to a NAT'd/private host to "just work" — from outside, that host has no reachable address
Forgetting NAT in allowlists — a whole office/VPC appears as ONE source IP, so "allow this user's IP" allows everyone behind that NAT

Pattern & benefit: NAT is what let a world with ~4 billion IPv4 addresses connect billions of devices — private ranges (topic 2) reused everywhere, translated to shared public IPs on the way out. Its outbound-easy/inbound-hard asymmetry explains cloud architecture (private subnets + NAT Gateway for egress, load balancers for ingress — topic 59) and a huge share of "can reach out but not in" tickets.

Gotcha — why it goes wrong: the directional asymmetry is the eternal confusion — a private host behind NAT can happily apt update and call APIs (outbound), yet nobody on the internet can reach its app (inbound), because it has no publicly reachable address. The fix is never "more NAT" — it's a load balancer, public IP, or port-forward (DNAT). And NAT collapses many hosts to one source IP, silently breaking per-user IP allowlists and rate limits.

09DNS — dig it, because it's always DNS★ used daily▶

Scenario: names → IPs. Every request starts here, and a shocking share of "the network is broken" incidents are actually DNS — stale caches, wrong records, slow resolvers. dig is how you see the truth.

🧠

Analogy: DNS is the internet's phone book, but distributed and cached at every level. Ask for api.shop.com and the question walks up a hierarchy — your resolver, then the .com operator, then shop.com's own nameservers — until someone answers with the number (IP). TTL is how long each cache trusts the answer before asking again, which is exactly why a DNS change "hasn't propagated yet."

Code

$ dig +short api.shop.com          # just the answer (the IP)
203.0.113.42
$ dig api.shop.com                 # full: ANSWER section, TTL, which server
;; ANSWER SECTION:
api.shop.com.  300  IN  A  203.0.113.42        ← 300 = TTL (5 min cache)

$ dig api.shop.com @8.8.8.8        # ask a SPECIFIC resolver (bypass local cache)
$ dig MX shop.com +short           # mail records; also: NS, TXT, CNAME, AAAA
$ dig +trace api.shop.com          # watch the FULL hierarchy walk (root → TLD → auth)

# the record types you'll meet:
#   A     name → IPv4          AAAA  name → IPv6
#   CNAME name → another name  MX    mail servers
#   TXT   arbitrary (SPF, verification)   NS  which nameservers are authoritative

Output — the "propagation" mystery, explained

$ dig api.shop.com @8.8.8.8 +short    → 203.0.113.42   (new IP)
$ dig api.shop.com @1.1.1.1 +short    → 198.51.100.9   (OLD IP — still cached!)
# different resolvers, different answers = a TTL that hasn't expired everywhere.
# "DNS hasn't propagated" is really "caches haven't expired yet" (wait out the TTL).

✅ Do — good use cases

dig +short name for a quick lookup; full dig to see TTL and the authoritative answer
dig name @resolver to compare resolvers when a change "hasn't propagated" — it's cache TTL, not magic
Lower a record's TTL before a planned IP change so the cutover is fast (raise it back after)

❌ Don't — anti-patterns

Blaming "the network" before ruling out DNS — resolve the name first (dig), THEN test the IP (topic 23)
Expecting instant DNS changes — the old answer lives in caches worldwide until its TTL expires; plan around it

Pattern & benefit: dig makes the invisible first step of every request visible — the name, the resolved IP, the TTL, and which server answered. Separating "can I resolve the name?" from "can I reach the IP?" (topic 23) instantly halves the search space of most connectivity problems, and the running joke "it's always DNS" is a joke because it's so often true.

Gotcha — why it goes wrong: TTL-based caching is the source of the "propagation" myth — DNS doesn't push changes, caches pull fresh answers only after the old TTL expires, so a change is live on some resolvers and stale on others for up to the TTL. Comparing dig @8.8.8.8 vs @1.1.1.1 reveals it. And CNAME chains add latency and can't coexist with other records at the same name — a subtle config trap.

10resolv.conf, /etc/hosts & the resolver order▶

Scenario: dig resolves a name fine but your app gets the wrong IP, or a name resolves that shouldn't exist. The host's own resolution config — /etc/hosts, /etc/resolv.conf, nsswitch — decides what your programs actually see.

🧠

Analogy: before your host phones the DNS phone book (topic 9), it checks its own little black book of personal contacts (/etc/hosts) — and a name written there wins, no matter what the public phone book says. /etc/resolv.conf is the phone number of the phone book itself (which DNS servers to call). Get either wrong and your apps resolve names differently than dig does.

Code

# /etc/hosts — static, LOCAL overrides (checked before DNS, usually):
127.0.0.1   localhost
10.0.5.99   db.internal        ← hardcode a name → IP (great for testing/overrides)

# /etc/resolv.conf — WHICH DNS servers this host asks (topic 9):
nameserver 10.0.0.2            ← the resolver to query
search      svc.cluster.local  ← suffixes auto-appended to bare names
options     ndots:5

# the ORDER is set by /etc/nsswitch.conf:
$ grep hosts /etc/nsswitch.conf
hosts:  files dns              ← "files" (/etc/hosts) FIRST, then dns

# why your app and dig can disagree:
$ dig +short db.internal       → (public DNS answer, or NXDOMAIN)
$ getent hosts db.internal     → 10.0.5.99   ← what the APP sees (respects /etc/hosts!)

Output — the tool that resolves like your app does

$ getent hosts db.internal
10.0.5.99  db.internal
# getent uses the SAME path as your programs (/etc/hosts THEN dns) —
# so it, not dig, tells you what the app actually resolves.

✅ Do — good use cases

getent hosts <name> to see what your apps resolve (respects /etc/hosts + nsswitch) — dig queries DNS only
/etc/hosts entries to pin or override a name for testing (point api.prod at staging locally)
Check /etc/resolv.conf when resolution is slow or wrong — the wrong/dead nameserver is a common culprit

❌ Don't — anti-patterns

Debugging app resolution with dig alone — it skips /etc/hosts, so it can disagree with reality; use getent
Leaving stale /etc/hosts overrides in place — a forgotten test entry silently misroutes an app for weeks

Pattern & benefit: understanding the resolution order (files → DNS, per nsswitch) resolves the maddening "dig works but my app connects to the wrong IP" class of bugs — because apps consult /etc/hosts first and dig never does. getent hosts is the tool that resolves exactly like your programs, making it the honest answer to "what does this host actually resolve this name to?"

Gotcha — why it goes wrong: dig and your application can genuinely disagree — dig talks straight to DNS, but your app (and curl, and ping) go through the system resolver, which checks /etc/hosts first. A stale hosts entry, a wrong resolv.conf nameserver, or the search/ndots settings (which quietly append suffixes and cause surprise lookups — a famous Kubernetes latency gotcha) all make apps resolve differently than dig suggests. Always confirm with getent.

The Web Stack

11HTTP anatomy — methods, status, headers★ used daily▶

Scenario: every API, webpage, and webhook is HTTP. Reading its methods, status codes, and headers fluently is the difference between guessing and knowing why a request behaves the way it does.

🧠

Analogy: an HTTP exchange is a formal letter and its reply. The method is your intent (GET = "please send me", POST = "here's something new"), the path is the recipient, headers are the envelope's metadata (who you are, what format you accept, your auth token), and the status code is the reply's opening line: 200 "here you go", 404 "no such person", 500 "our office caught fire", 401 "who are you?".

Code — a request and its parts

$ curl -v https://api.shop.com/orders/42
> GET /orders/42 HTTP/2                 ← method + path
> host: api.shop.com                    ← which site (topic 14 vhosts)
> authorization: Bearer eyJ...          ← auth header
> accept: application/json              ← "reply in this format"
< HTTP/2 200                            ← status: success
< content-type: application/json
< cache-control: max-age=60             ← cache me for 60s

# status code families — the FIRST digit is the story:
2xx  success        200 OK  201 Created  204 No Content
3xx  redirect       301 moved permanently  304 not modified (cached)
4xx  YOUR fault     400 bad request  401 unauth  403 forbidden  404 not found  429 rate-limited
5xx  SERVER fault   500 internal  502 bad gateway  503 unavailable  504 gateway timeout

The 502/503/504 distinction — critical for ops

502 Bad Gateway      → proxy reached the backend, got garbage/none (backend crashed?)
503 Unavailable      → proxy has NO healthy backend (all down / overloaded)
504 Gateway Timeout  → backend accepted but didn't reply in time (slow query? topic 48)
# these three are usually the PROXY (topic 14) reporting on your app, not the proxy itself.

✅ Do — good use cases

Read the status-code family first: 4xx = the request is wrong, 5xx = the server is wrong — that alone halves the search
Distinguish 502/503/504 — they point at backend-crashed vs no-backends vs backend-too-slow (topic 14)
Inspect headers with curl -v/-I — auth, content-type, caching, and redirects are all right there

❌ Don't — anti-patterns

Treating all 5xx as "server broken" without reading which — 504 is a timeout (often a slow DB query), a different fix than 502
Ignoring 4xx as "client problem, not mine" — a 401/403 often means your auth header or config is wrong

Pattern & benefit: HTTP is a text protocol you can read with your eyes (curl -v) — method, headers, and status make every request self-describing. The status-code families are a diagnostic shortcut used everywhere from browser devtools to load-balancer logs to the DevOps playbook's ALB metrics (EKS topic 40): first digit tells you whose fault, the rest tells you what.

Gotcha — why it goes wrong: the 502/503/504 trio confuses teams because they're emitted by the proxy (nginx/ALB, topic 48) about your backend — so a 504 isn't "the load balancer is slow," it's "your app didn't answer in time" (often a slow database query, DevOps playbook territory). Reading which 5xx you got points at the right layer immediately, instead of restarting the proxy that was only the messenger.

12TLS & certificates — how HTTPS trusts★ used daily▶

Scenario: the padlock, cert expiry pages, "certificate not trusted" errors, and TLS termination at a load balancer — HTTPS is HTTP wrapped in TLS, and the cert is how the client knows it's really talking to who it thinks.

🧠

Analogy: a TLS certificate is a passport issued by a trusted authority. When you connect, the server shows its passport (cert), your client checks it was issued by an authority it trusts (a Certificate Authority in its trust store), that it's for this name (shop.com), and that it hasn't expired. Only then do the two agree on a secret language (the encrypted session) nobody eavesdropping can read. A self-signed cert is a hand-drawn passport — real to you, trusted by no one else.

Code — inspect and verify certs

# the handshake, then encrypted HTTP inside. Inspect the cert offered:
$ echo | openssl s_client -connect shop.com:443 -servername shop.com 2>/dev/null \
    | openssl x509 -noout -subject -issuer -dates
subject=CN = shop.com
issuer=C = US, O = Let's Encrypt, CN = R3        ← who vouched for it
notBefore=Jul  1 00:00:00 2026 GMT
notAfter=Sep 29 23:59:59 2026 GMT                ← EXPIRY (the 3am page)

$ curl -vI https://shop.com          # curl verifies the chain for you
*  SSL certificate verify ok.        ← trusted, in-date, name matches
$ curl https://self-signed.local
curl: (60) SSL certificate problem: self signed certificate   ← not in trust store

# -servername matters: one IP can host many certs (SNI) — topic 48

Output — the three ways a cert fails

EXPIRED        notAfter in the past → "your connection is not private" (renew it!)
NAME MISMATCH  cert says shop.com, you asked api.shop.com → wrong cert / missing SAN
UNTRUSTED      issuer not in the client's CA store → self-signed or private CA

✅ Do — good use cases

Automate renewal (Let's Encrypt/certbot, or ACM in cloud) — expiry is the most common, most preventable outage
Monitor notAfter with alerts weeks ahead — don't learn about expiry from users
Terminate TLS at the load balancer/proxy (topic 13/14, DevOps playbook EKS topic 6) — central certs, offloaded crypto

❌ Don't — anti-patterns

curl -k / disabling verification to "fix" a cert error in production — you've turned off the entire point of TLS
Manual cert renewal on a calendar reminder — humans forget; the expiry outage is legendary and 100% avoidable

Pattern & benefit: TLS gives you encryption and authentication (you're talking to the real server) in one handshake, and because it wraps HTTP transparently, your app code doesn't change. Terminating it at the edge (LB/proxy) centralizes certificate management and moves crypto off your app servers — the standard pattern the DevOps playbook automates with ACM (EKS topic 40).

Gotcha — why it goes wrong: certificate expiry is the single most common self-inflicted outage on the internet — a cert quietly reaches notAfter and every client suddenly refuses to connect. Automate renewal and alert early. The other two: name mismatch (the cert's SANs don't include the name you requested — common with wildcards and multi-domain setups) and untrusted issuer (self-signed or an internal CA not in the client's trust store) — three distinct failures, three distinct fixes, all reported as "certificate problem."

13Load balancing — L4 vs L7, and health checks▶

Scenario: one server can't handle the traffic (or can't be a single point of failure), so requests spread across many. Load balancers do the spreading — and the L4-vs-L7 choice decides what they can do.

🧠

Analogy: a load balancer is the host at a busy restaurant's door, seating diners across many tables (servers). An L4 host just counts heads and points ("table 3, go") — fast, knows nothing about the order. An L7 host reads the reservation ("vegetarian party → the veg-kitchen section, VIP → the good tables") — it understands the content (URLs, headers) and routes smartly. Both quietly stop seating diners at a table where the waiter collapsed (health checks).

Code

# L4 (transport): routes by IP:port, blind to content. Fast, any protocol.
#   AWS NLB, HAProxy TCP mode. Good for databases, gRPC, raw TCP.
# L7 (application): reads HTTP — routes by host/path/header, does TLS, retries.
#   AWS ALB, nginx, HAProxy HTTP mode. Good for web/APIs.

# the feature that makes it all work — HEALTH CHECKS:
#   LB polls each backend's /health every few seconds.
#   fails N times → pulled from rotation (no traffic) → passes → back in.
#   THIS is why a rolling deploy (DevOps playbook K8s topic 20/21) is zero-downtime.

$ curl -H "Host: api.shop.com" http://lb-address/health   # what the LB checks
{"status":"ok"}
# algorithms: round-robin (default), least-connections, IP-hash (sticky sessions)

L4 vs L7 — pick by what you need to route on

              L4 (NLB / TCP)            L7 (ALB / nginx / HTTP)
sees          IP + port only            full HTTP: host, path, headers, cookies
can route by  —                         /api → svc-a, /img → svc-b, host-based
can do        any protocol, fastest     TLS termination, retries, rewrites, WAF
use for       databases, gRPC, non-HTTP web apps, APIs, multi-service ingress

✅ Do — good use cases

L7 (ALB/nginx) for web/APIs — host/path routing, TLS termination (topic 12), one entry for many services
L4 (NLB) for non-HTTP or when you need raw speed and protocol transparency (databases, gRPC)
Meaningful health-check endpoints (/health that checks real readiness) — they gate every deploy and failover

❌ Don't — anti-patterns

A health check that always returns 200 regardless of real state — the LB keeps sending traffic to broken backends
L7 features (path routing, header inspection) expected from an L4 balancer — it can't see inside the packets

Pattern & benefit: load balancers deliver both scale (spread load across N servers) and availability (health checks route around failures) — and the health-check mechanism is precisely what makes rolling deploys and node failures invisible to users (the exact machinery behind DevOps playbook K8s topics 20, 21 and EKS topic 40). L4 vs L7 is the "how smart does the routing need to be?" decision.

Gotcha — why it goes wrong: the health check is the whole game, and a shallow one is the classic trap — if /health returns 200 without checking the database/dependencies the app actually needs, the LB happily sends traffic to instances that then 500 every real request (a "healthy" pool serving errors). Conversely, a health check that's too strict (pings the DB, which blips) can pull every backend at once — the same self-inflicted-outage warning as liveness probes (topic 20 / DevOps playbook).

14Reverse proxies — nginx as the front door★ used daily▶

Scenario: in front of your app sits a reverse proxy (nginx, most often) doing TLS, routing, caching, and buffering. Understanding it explains vhosts, upstreams, and most 502/504 errors (topic 11).

🧠

Analogy: a reverse proxy is the receptionist for a building full of teams. Visitors (clients) only ever talk to reception; it checks IDs (TLS), reads the request ("Marketing? third floor" — routes by host/path to the right upstream app), handles the crowd so the teams aren't mobbed (buffering, rate-limiting), and keeps common answers on the desk (caching). The app servers stay hidden and safe on their private floors.

nginx — the config shape you'll read constantly

server {
    listen 443 ssl;
    server_name api.shop.com;              # vhost: match by Host header (topic 11)
    ssl_certificate     /etc/ssl/shop.crt; # TLS terminates HERE (topic 12)
    ssl_certificate_key /etc/ssl/shop.key;

    location /api/ {
        proxy_pass http://app_backend;     # forward to the upstream app
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;   # preserve real client IP!
        proxy_read_timeout 30s;            # too short → 504 (topic 11)
    }
    location /static/ { root /var/www; }   # serve files directly, skip the app
}
upstream app_backend {                     # the pool (basic L7 LB — topic 47)
    server 10.0.5.20:3000;
    server 10.0.5.21:3000;
}
$ nginx -t && nginx -s reload             # TEST config, then reload with no downtime

Output — why your app sees the wrong client IP

Without X-Forwarded-For: app logs show 10.0.5.1 (the PROXY) for every request.
With it: app reads X-Forwarded-For → the REAL client IP.
# every "all my traffic appears to come from one IP" bug = missing/!trusted XFF header.

✅ Do — good use cases

nginx -t before every reload — it validates config and reloads with zero dropped connections
Forward X-Forwarded-For/X-Forwarded-Proto and trust them in the app — or you lose real client IPs and HTTPS awareness
Serve static files from the proxy, proxy only dynamic requests to the app — faster and lighter

❌ Don't — anti-patterns

Blaming the proxy for 502/504 — they usually mean the upstream app crashed (502) or was too slow (504, raise proxy_read_timeout AND fix the slow endpoint)
Reloading with an untested config — a syntax error can take the front door down; always nginx -t first

Pattern & benefit: the reverse proxy centralizes the cross-cutting concerns — TLS termination (topic 12), routing by host/path (vhosts), rate limiting, caching, static files, real-client-IP forwarding — so your app stays simple and hidden. It's the same role the DevOps playbook's Ingress/ALB plays in Kubernetes (EKS topic 40): one smart front door for many backends.

Gotcha — why it goes wrong: two evergreen proxy bugs. (1) Lost client IP: without forwarding and trusting X-Forwarded-For, your app sees the proxy's IP for every request, breaking geolocation, rate limits, and logs. (2) Timeout mismatch: proxy_read_timeout shorter than a legitimately slow endpoint yields a 504 while the backend is still working — but the real fix is usually the slow query (topic 11's point), not just a bigger timeout. And always nginx -t: a typo'd reload can shut the front door.

Debugging the Network

15ping & ICMP — is it even alive?★ used daily▶

Scenario: the first question in any connectivity problem — is the host reachable at all? ping answers it at the IP layer (topic 1), before you worry about ports, DNS, or the app.

🧠

Analogy: ping is knocking on the door and listening for a knock back. It tells you the house exists, someone's home at the address, and roughly how far away it is (round-trip time) — but nothing about whether the shop inside is open (that's the port/app, topics 41/45). A quick knock, the cheapest possible reachability test.

Code

$ ping -c4 shop.com          # -c4 = 4 packets then stop (not forever)
PING shop.com (203.0.113.42): 56 data bytes
64 bytes from 203.0.113.42: icmp_seq=0 ttl=54 time=12.3 ms   ← alive, 12ms away
64 bytes from 203.0.113.42: icmp_seq=1 ttl=54 time=11.9 ms
--- shop.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, avg 12.1ms

# what the results mean:
#   replies + low time      → reachable, healthy latency (topic 24)
#   "Request timeout"       → no reply: host down, firewall, or ICMP blocked (see gotcha)
#   partial loss (2/4)      → intermittent — congestion or a flaky link
#   "Name or service..."    → DNS failed, not a reachability issue (topic 9)!
$ ping -c1 10.0.5.1          # ping the GATEWAY to isolate local vs remote (topic 4)

Output — ping tells you WHICH problem you have

ping shop.com  → "Name or service not known"   → it's DNS (topic 9), stop here
ping 203.0.113.42 → replies                      → IP is reachable; problem is higher up
ping 203.0.113.42 → 100% loss                    → unreachable OR ICMP blocked (topic 5)
# ping the name AND the IP: if name fails but IP works, you've found DNS.

✅ Do — good use cases

ping -c4 as step one of any connectivity check — cheapest "is it alive?" test
Ping the name AND the raw IP — name-fails-but-IP-works pinpoints DNS instantly (topic 9)
Ping the gateway (topic 4) to split "my local network" from "the path beyond"

❌ Don't — anti-patterns

Concluding "host is down" from ping timeout alone — many hosts/firewalls (and most cloud security groups) block ICMP while TCP works fine
Reading a successful ping as "the service works" — ping proves the host answers, not that the port/app does (topics 5, 7)

Pattern & benefit: ping isolates the IP layer in one command — reachable or not, and how far (latency, topic 58) — which is exactly the bottom rung of the debug-in-layers ladder (topic 23). Pinging both the name and the IP is the fastest DNS-vs-network split there is.

Gotcha — why it goes wrong: ICMP is frequently blocked — AWS security groups, corporate firewalls, and hardened hosts often drop ping while happily serving TCP, so a ping timeout does NOT prove the host is down (test the actual port with nc -zv, topic 39). And ping resolving the name is itself a DNS test: "Name or service not known" is a resolver problem (topic 9), a completely different fix than "host unreachable."

16traceroute & mtr — where does the path break?▶

Scenario: a host is unreachable or slow, and you need to know where along the path it fails — your network, your ISP, or somewhere in the middle. traceroute maps the hops; mtr watches them live.

🧠

Analogy: traceroute is sending a series of scouts down a road, each told to stop one town further and report back where they are. Scout 1 stops at the first town and radios its name; scout 2 reaches the second; and so on — building a map of every town (router) between you and the destination. When the scouts stop coming back at town 5, you've found where the road is blocked.

Code

$ traceroute shop.com        # each line = one hop (router) toward the target
 1  10.0.5.1        1.2 ms      ← your gateway (topic 4)
 2  100.64.0.1      3.4 ms      ← your ISP
 3  72.14.x.x       11 ms       ← upstream
 4  * * *                       ← this hop didn't reply (often normal — see gotcha)
 5  203.0.113.42    12 ms       ← destination reached ✓

# mtr = traceroute + ping, continuously — the better tool:
$ mtr shop.com               # live-updating, shows LOSS% per hop
  Host                    Loss%   Avg  Best  Wrst
  1. 10.0.5.1              0.0%   1.1   0.9   2.1
  2. 100.64.0.1            0.0%   3.2   2.8   9.4
  5. 203.0.113.42         18.0%  142   12   890   ← loss + latency SPIKE starts here
$ mtr --report -c 100 shop.com    # 100 samples, one-shot report for tickets

Output — reading where the trouble starts

Loss/latency that STARTS at a hop and continues → the problem is at/after that hop.
Loss at ONE middle hop but fine after → that router just deprioritizes ICMP (not a real problem).
Path dies at hop 2 (your ISP) → not your fault, not the destination's — escalate to the ISP.

✅ Do — good use cases

mtr over plain traceroute — continuous sampling reveals intermittent loss a single traceroute misses
Locate blame: loss starting at your gateway (you), your ISP (them), or the far end (the destination)
mtr --report output in tickets — it's the evidence ISPs and providers actually act on

❌ Don't — anti-patterns

Panicking over * * * or loss at a single middle hop — routers often rate-limit ICMP replies while forwarding real traffic fine
Reading latency at intermediate hops as user-facing latency — only the final hop's RTT is what your users experience

Pattern & benefit: traceroute/mtr turn "somewhere out there it's broken" into "loss begins at hop 5, which is my ISP's router" — assigning the problem to a specific party (you, your ISP, transit, or the destination). mtr's continuous per-hop loss is the single best tool for intermittent, hard-to-catch network flakiness.

Gotcha — why it goes wrong: the biggest misread is treating loss/timeouts at a middle hop as the culprit — many routers deprioritize or rate-limit the ICMP that traceroute relies on, so they show loss while forwarding your actual traffic perfectly. What matters is loss that starts at a hop and persists to the destination, and the final hop's numbers. A single starred middle hop with a clean final hop is almost always a non-issue.

17curl mastery — the Swiss army knife of HTTP★ used daily▶

Scenario: test an API, debug a redirect, time a slow request, send auth headers, hit a specific backend — curl does it all, and reading its verbose output is a core daily skill.

🧠

Analogy: curl is a programmable, brutally honest web browser with no chrome — it makes exactly the request you specify and shows you exactly what came back, headers and all, no rendering, no hidden retries, no cache guessing. When a browser "works but the API doesn't" (or vice versa), curl is how you see the raw truth of the exchange.

Code — the flags you'll use forever

$ curl -v https://api.shop.com/x     # -v = show request + response HEADERS (topic 11)
$ curl -I https://shop.com           # -I = HEADERS ONLY (HEAD request)
$ curl -L http://shop.com            # -L = FOLLOW redirects (301/302)
$ curl -s -o /dev/null -w "%{http_code}\n" https://shop.com   # just the status code

# sending data & auth:
$ curl -X POST -H "Content-Type: application/json" \
       -H "Authorization: Bearer $TOKEN" \
       -d '{"name":"mug"}' https://api.shop.com/products

# the debugging power moves:
$ curl -w "@curl-format.txt" -o /dev/null -s https://shop.com   # TIMING breakdown
$ curl --resolve shop.com:443:10.0.5.20 https://shop.com        # test ONE backend
#        └ bypass DNS: "pretend shop.com = this IP" (topic 10) — test a specific server
$ curl -k https://self-signed.local   # skip cert check (DEBUG ONLY — topic 46)

Output — the timing breakdown that finds the slow part

$ curl -w "dns: %{time_namelookup}  connect: %{time_connect}  tls: %{time_appconnect}  ttfb: %{time_starttransfer}  total: %{time_total}\n" -so /dev/null https://shop.com
dns: 0.004  connect: 0.031  tls: 0.089  ttfb: 1.240  total: 1.245
#                                              └ TTFB huge, rest tiny → the SERVER is slow
#                                                (not DNS, not network) — go look at the app.

✅ Do — good use cases

curl -v to see the real request/response headers; -w timing to find WHICH phase is slow (DNS/connect/TLS/TTFB)
--resolve name:port:IP to test one specific backend behind a load balancer or DNS (topic 10/13)
Scriptable checks: -s -o /dev/null -w "%{http_code}" for health checks and CI smoke tests

❌ Don't — anti-patterns

-k (skip cert verification) anywhere but local debugging — it disables TLS's entire security guarantee (topic 12)
Debugging "the site is slow" without the -w timing breakdown — it tells you if it's DNS, TLS, or the server, saving hours of guessing

Pattern & benefit: curl is the one tool spanning the whole web stack — DNS (topic 9), TLS (topic 12), HTTP (topic 11), and backend selection (topic 13) — and its -w timing breakdown localizes "slow" to a specific phase in one command. It's the scriptable, honest client behind most health checks, smoke tests, and API debugging.

Gotcha — why it goes wrong: the huge time-saver people miss is the -w timing breakdown — time_namelookup vs time_connect vs time_appconnect (TLS) vs time_starttransfer (TTFB) tells you instantly whether "slow" is DNS, TCP, TLS, or the server thinking. Without it, teams blame "the network" for what the breakdown reveals is a slow database query (high TTFB, everything else fast). And --resolve is the clean way to test one backend without editing /etc/hosts (topic 10).

18tcpdump — the packet wiretap▶

Scenario: when higher-level tools disagree or lie, drop to the wire. tcpdump captures the actual packets — the ground truth of what left and arrived — for when "curl says X but the app says Y."

🧠

Analogy: if strace (topic 16) wiretaps a program's calls to the kernel, tcpdump wiretaps the wire itself — every packet in and out, unfiltered by any application's opinion of what happened. It's the security-camera footage of the network: when accounts conflict, you go watch the tape of what actually crossed the cable.

Code

# capture is cheap; the ART is FILTERING (or you drown):
$ sudo tcpdump -i any -n port 5432          # -n = no DNS lookups; only :5432 traffic
$ sudo tcpdump -i eth0 host 10.0.5.99       # only to/from one host
$ sudo tcpdump -i any 'port 443 and host shop.com'
$ sudo tcpdump -i any -c 20 port 80          # -c 20 = stop after 20 packets

# see the TCP handshake (topic 5) live — SYN, SYN-ACK, ACK:
$ sudo tcpdump -i any -n 'tcp port 5432 and (tcp[tcpflags] & tcp-syn != 0)'
10.0.5.20.44321 > 10.0.5.99.5432: Flags [S]      ← SYN (client trying)
# ...if you see [S] repeated with NO [S.] reply → firewall dropping (topic 5/20)

# capture to a file, analyze in Wireshark later:
$ sudo tcpdump -i any -w capture.pcap port 443
$ sudo tcpdump -r capture.pcap -n            # read it back

Output — the "timed out" mystery, solved on the wire

$ sudo tcpdump -i any -n host db.prod and port 5432
10.0.5.20.51234 > 10.0.8.44.5432: Flags [S]     ← we sent SYN
10.0.5.20.51234 > 10.0.8.44.5432: Flags [S]     ← retry... no reply
10.0.5.20.51234 > 10.0.8.44.5432: Flags [S]     ← retry... still nothing
# SYNs going out, ZERO replies coming back = a firewall is DROPPING us (topic 5,20).
# not the app, not DNS — the packets never get an answer. Case closed.

✅ Do — good use cases

Always filter (host, port, -c N) — an unfiltered capture on a busy host is an unreadable firehose
Confirm ground truth: are packets leaving? arriving? getting replies? — settles app-vs-network disputes
Capture to -w file.pcap and open in Wireshark for deep analysis (TLS handshakes, retransmits, resets)

❌ Don't — anti-patterns

Running tcpdump with no filter on a production host — the volume overwhelms you (and can add load); scope it tightly
Reaching for tcpdump first — exhaust ping/ss/curl (topics 15/7/17) before the wiretap; it's the last resort, not the first

Pattern & benefit: tcpdump is the final arbiter — when every higher tool gives conflicting stories, the packets don't lie. Seeing SYNs leave with no reply (firewall drop), or replies arriving that the app claims it never got (app bug), ends debates instantly. It's the network twin of strace (topic 16): the ground-truth wiretap for when logs and tools disagree.

Gotcha — why it goes wrong: two traps. (1) No filter = drowning: on a busy host an unfiltered capture scrolls faster than you can read and adds overhead — always constrain by host/port/-c. (2) Forgetting -n makes tcpdump do a DNS lookup per packet, which both slows it and pollutes your capture with DNS traffic (and can hang if DNS is the very thing that's broken). The reading skill — SYN with no SYN-ACK = dropped (topic 5) — is what turns raw packets into a diagnosis.

19TCP states & TIME_WAIT — the socket lifecycle▶

Scenario: ss shows thousands of connections in TIME_WAIT, or a server "runs out of ports" under load. TCP connections have a lifecycle, and its states explain both — and prevent a panic over what's usually normal.

🧠

Analogy: TIME_WAIT is the mandatory cooling-off period after a phone call ends — the line isn't reused for a couple of minutes, in case a late "goodbye" packet from the old call is still in transit and would confuse a brand-new call reusing the same numbers. Thousands of them isn't a leak; it's thousands of recently, cleanly closed calls still in their cooldown. Alarming to see, usually harmless.

Code

# the lifecycle (see it with ss -tan):
#   LISTEN → SYN-SENT/RECV → ESTABLISHED → (close) → TIME_WAIT → gone
$ ss -tan state time-wait | wc -l         # count them
$ ss -s                                    # summary of all states
Total: 4821
TCP:   4102 (estab 210, timewait 3801, ...)   ← lots of TIME_WAIT

# TIME_WAIT is on the side that CLOSED FIRST (usually the client).
# it lasts ~60s (2×MSL) then vanishes on its own. Normal for busy clients.

# the ACTUAL problem it can cause — ephemeral port exhaustion:
$ sysctl net.ipv4.ip_local_port_range      # the pool of outbound ports
32768 60800                                 # ~28k — a client making MANY short
                                            # connections can exhaust these
# the REAL fix (not tcp_tw_recycle — that's dangerous/removed):
#   → reuse connections (HTTP keep-alive, connection pools) — topic 45
#   → connect via fewer, longer-lived connections instead of many short ones

Output — the states and what they mean

ESTABLISHED  active, carrying data — the healthy state
TIME_WAIT    recently closed (by this side), cooling down ~60s — usually FINE
CLOSE_WAIT   the OTHER side closed, but YOUR app hasn't — a pile of these = APP BUG
             (your code isn't closing sockets — the real leak to worry about)

✅ Do — good use cases

Recognize a pile of TIME_WAIT as normal for busy clients — it self-clears in ~60s
Fix port exhaustion the right way: connection reuse (keep-alive, pooling — topic 45), not risky sysctl hacks
Worry about CLOSE_WAIT instead — many of those means YOUR app is leaking un-closed sockets (topic 15)

❌ Don't — anti-patterns

Enabling net.ipv4.tcp_tw_recycle to "fix" TIME_WAIT — it breaks NAT'd clients (topic 8) and was removed from modern kernels for good reason
Confusing TIME_WAIT (normal, self-clearing) with CLOSE_WAIT (your app bug) — opposite meanings, opposite actions

Pattern & benefit: knowing the TCP state machine stops a common false alarm (TIME_WAIT is healthy) and points at the real one (CLOSE_WAIT = your app isn't closing sockets, an fd leak per topic 15). It also explains why connection reuse — keep-alive, pooling — matters so much under load: it avoids the constant open/close churn that fills the ephemeral port pool.

Gotcha — why it goes wrong: the internet is full of advice to enable tcp_tw_recycle to clear TIME_WAIT — don't: it silently breaks connectivity for clients behind NAT (topic 8) and was removed from Linux 4.12+ entirely. TIME_WAIT is working as designed; the fix for its symptoms is fewer, longer-lived connections. And the truly worrying state is CLOSE_WAIT — a growing pile means your application accepted a close from the peer but never called close() itself, leaking file descriptors (topic 15) until it hits the limit.

20iptables & nftables — the host firewall▶

Scenario: "the port is open, the app is listening (topic 7), but I still can't connect" — often a firewall silently dropping packets. Understanding the host firewall explains those timeouts (topic 5) and how to inspect the rules.

🧠

Analogy: the firewall is a bouncer with a checklist, reading it top to bottom for every packet. The first matching rule wins — "allow SSH from the office," "allow web from anyone," and finally the house policy "everyone else: turn away." A DROP is the silent bouncer who just ignores you (you time out, topic 39); a REJECT is the one who says "no" to your face (instant connection refused). Order matters: a "deny all" rule placed too high blocks everything below it.

Code

# inspect current rules (the FIRST thing to do — don't guess):
$ sudo iptables -L -n -v                  # -n numeric, -v verbose (packet counts!)
Chain INPUT (policy DROP)                  ← default: drop anything not allowed
 pkts target  prot source        destination  ports
 8231 ACCEPT  tcp  10.0.5.0/24   0.0.0.0/0    tcp dpt:22    ← SSH from subnet
45102 ACCEPT  tcp  0.0.0.0/0     0.0.0.0/0    tcp dpt:443   ← HTTPS from anywhere
    0 ACCEPT  tcp  0.0.0.0/0     0.0.0.0/0    tcp dpt:5432  ← 0 packets... blocked upstream?

# nftables — the modern replacement (same concepts, cleaner syntax):
$ sudo nft list ruleset

# cloud note: SECURITY GROUPS are firewalls too, OUTSIDE the host —
# a host with no iptables rules can still be blocked by its security group (topic 25)!
$ sudo iptables -L -n | grep 5432          # is the app port even allowed?

Output — DROP vs REJECT decides your symptom (topic 5)

rule: DROP    → client sees "Connection timed out"  (silent — packet vanishes)
rule: REJECT  → client sees "Connection refused"    (explicit RST sent back)
# so: "timed out" often = firewall DROP;  "refused" = nothing listening OR a REJECT.
# the packet-count column (-v) shows which rules are actually being hit.

✅ Do — good use cases

iptables -L -n -v (packet counts) when a listening port (topic 7) still times out — the firewall is the prime suspect
Remember rules match top-to-bottom, first-match-wins — order is as important as the rules themselves
In cloud, check BOTH the host firewall AND the security group (topic 25) — either can silently block you

❌ Don't — anti-patterns

Flushing rules (iptables -F) on a remote box with a default DROP policy — you can lock yourself out mid-command
Assuming "no host firewall rules" = "not firewalled" in the cloud — the security group outside the host may be dropping you

Pattern & benefit: the firewall closes the loop on the topic 39 mystery — a DROP rule is exactly why a reachable host with a listening service still "times out," while REJECT gives "refused." Reading the rules (with packet counts) tells you whether traffic is even reaching your service, and the same first-match-wins model scales up to cloud security groups and Kubernetes NetworkPolicies (DevOps playbook K8s topic 32).

Gotcha — why it goes wrong: the DROP-vs-REJECT distinction ties directly to topic 39's refused-vs-timeout: a firewall DROP makes the client hang until timeout (the packet silently disappears), while REJECT sends back a rejection for an instant "refused." So "times out" strongly implies a firewall drop somewhere, not a down service. And the cloud gotcha bites hardest — engineers check iptables, find it empty, and conclude "not firewalled," missing that the AWS security group or NACL (topic 25) — a firewall outside the host — is doing the dropping.

21ssh mastery & tunnels — the remote-access power tool★ used daily▶

Scenario: SSH is how you reach every server, but it's far more than a remote shell — keys, config shortcuts, jump hosts, and tunnels that reach services you can't touch directly.

🧠

Analogy: SSH is an encrypted private corridor you build on demand between two buildings. Normally you walk it to reach a remote office (shell). But you can also run pneumatic tubes through the corridor (tunnels): a local port on your desk that secretly pops out inside the remote building next to a locked-away service (a database on a private network), letting you reach it as if it were on your own desk.

Code

# keys, not passwords (topic 4: the key MUST be chmod 600):
$ ssh-keygen -t ed25519              # generate a modern key
$ ssh-copy-id user@server           # install your public key there

# ~/.ssh/config — stop typing long commands:
#   Host prod
#     HostName 203.0.113.42
#     User deploy
#     IdentityFile ~/.ssh/prod_ed25519
#     ProxyJump bastion              ← auto-hop through a jump host!
$ ssh prod                          # now just this

# the killer feature — LOCAL port forward (reach a private DB):
$ ssh -L 5432:db.internal:5432 bastion
#        └local└──remote target──┘ └through this host
$ psql -h localhost -p 5432         # your laptop → tunnel → private db.internal!

$ ssh -D 1080 bastion               # SOCKS proxy — browse AS the bastion
$ scp file prod:/tmp/               # copy over ssh;  rsync -e ssh (topic 27)

Output — the tunnel reaching an unreachable service

$ ssh -L 5432:db.internal:5432 bastion    # db.internal has NO public IP (topic 8)
$ psql -h localhost -p 5432 -U app
psql (16.4)  →  app_prod=>                 ← connected to a private DB, via the tunnel.
# your packets: laptop:5432 → [encrypted ssh] → bastion → db.internal:5432

✅ Do — good use cases

Keys over passwords everywhere; ~/.ssh/config with ProxyJump for bastion/jump-host hops
ssh -L tunnels to reach private databases/dashboards behind a bastion (topic 8's "inbound is hard")
Key at chmod 600 (topic 4) — SSH refuses loose-permissioned keys by design

❌ Don't — anti-patterns

Password auth or shared keys on production — per-person keys give per-person audit and revocation
Leaving long-lived tunnels open as ad-hoc infrastructure — they're for debugging, not a substitute for proper access (VPN/LB)

Pattern & benefit: SSH is remote shell, file transfer (scp/rsync — topic 27), jump-host chaining, and port tunneling in one tool — and the tunnel is the pragmatic answer to NAT's inbound problem (topic 8): you can't route to a private service, but you can build an encrypted corridor out from it. ~/.ssh/config turns all of it into short, memorable commands.

Gotcha — why it goes wrong: the most common SSH snag is key permissions — a private key that's group/world-readable (topic 4) makes SSH silently refuse it with "permissions are too open," falling back to password or failing; chmod 600 fixes it. And tunnel direction confuses people: -L (local forward) opens a port on your machine that reaches a service on the remote side — the common case; -R (remote) is the reverse and rarely what you want. Read -L local:target:port as "expose target:port on my local port."

22ip addr & interfaces — the modern net toolkit▶

Scenario: what's this host's IP? which interface? is the link even up? The ip command (replacing ifconfig) is the single tool for addresses, links, and routes (topic 4).

🧠

Analogy: if the host is a building, its network interfaces are the doors to the outside — eth0 the front door, lo the internal intercom (localhost), docker0/veth the doors to the container annex. ip is the facilities manager's clipboard: which doors exist, whether each is open (UP) or bolted (DOWN), and what address is painted on each.

Code — one tool, three jobs

$ ip -c addr                 # (or: ip a) — interfaces + their IP addresses
1: lo:    <LOOPBACK,UP> inet 127.0.0.1/8              ← localhost (topic 7)
2: eth0:  <BROADCAST,UP,LOWER_UP> inet 10.0.5.20/24  ← the real address (topic 2)
3: docker0: inet 172.17.0.1/16                        ← container bridge (topic 34)

$ ip link                    # just the interfaces + UP/DOWN state + MAC (topic 3)
$ ip route                   # the routing table (topic 4)
$ ip neigh                   # the ARP cache (topic 3)

# the state words that matter:
#   UP + LOWER_UP  → interface enabled AND cable/carrier present (good)
#   UP, NO-CARRIER → enabled but no physical link (unplugged / down switch port)
#   DOWN           → administratively disabled
$ ip link set eth0 up        # bring an interface up
$ ip -s link show eth0       # -s = stats: RX/TX packets, errors, drops

Output — the old vs new commands

OLD (deprecated)      NEW (use these)
ifconfig              ip addr
route -n              ip route
arp -a                ip neigh
netstat -tlnp         ss -tlnp   (topic 7)
# the net-tools are often not even installed on modern minimal images.

✅ Do — good use cases

ip a to find this host's address/interface; ip link to check UP/DOWN and carrier
Learn ip (addr/link/route/neigh) — it's the one modern tool and it's on every current distro
ip -s link for interface error/drop counters when a NIC is flaky

❌ Don't — anti-patterns

Relying on ifconfig/route/netstat — deprecated and frequently absent from minimal images/containers
Confusing UP (admin-enabled) with LOWER_UP (carrier present) — an interface can be UP with no cable (NO-CARRIER)

Pattern & benefit: the ip suite unifies what used to be four separate legacy tools (ifconfig/route/arp/netstat) into one consistent command covering addresses (topic 2), links (topic 3), and routes (topic 4) — the layers 2–3 toolkit in one place. Knowing it means you're productive on any modern box, including the minimal container images where the old tools don't exist.

Gotcha — why it goes wrong: muscle memory for ifconfig fails on modern minimal systems — it's deprecated and often not installed, so scripts and habits built on net-tools break on fresh containers/distros; the ip equivalents always work. And the UP-vs-LOWER_UP distinction catches people: an interface shown as UP is administratively enabled, but NO-CARRIER means no cable/link — "the interface is up but there's no connectivity" is a physical-layer (topic 1) problem, not a config one.

23The "is it down?" runbook — debug in layers★ used daily▶

Scenario: "the service is unreachable" — the most common network page. Instead of guessing, walk the layers (topic 1) bottom-up: DNS → reachability → port → TLS → app. Each step eliminates half the search space.

🧠

Analogy: diagnosing a dead phone line, one segment at a time: is the number in the directory (DNS)? does the building answer a knock (ping)? is that specific office staffed (port)? do they speak your language (TLS)? and finally, do they actually help you (the app)? You don't suspect the office staff before confirming the building even exists — you check each segment in order, and the first broken one is your answer.

The runbook — run these in order, stop at the first failure

# 1. DNS — does the name resolve? (topic 9/10)
$ dig +short api.shop.com          → no answer?  → DNS problem. STOP.
$ getent hosts api.shop.com        → what the APP actually resolves (topic 10)

# 2. REACHABILITY — is the host up at the IP layer? (topic 15)
$ ping -c3 <the-resolved-IP>        → (ICMP may be blocked — don't over-trust)
$ mtr <IP>                          → where does the path break? (topic 16)

# 3. PORT — is the service listening & reachable? (topic 5/7/20)
$ nc -zv <IP> 443                   → "refused" = nothing there / "timeout" = firewall
$ ss -tlnp | grep :443             → (ON THE SERVER) is it even bound? (topic 7)

# 4. TLS — does the handshake & cert check pass? (topic 12)
$ curl -vI https://api.shop.com    → cert expired? name mismatch?

# 5. APP — does it actually respond correctly? (topic 11/17)
$ curl -v https://api.shop.com/health  → 200? 502 (backend down)? 504 (slow)?
$ curl -w timing... (topic 17)          → which phase is slow?

Output — the layer that fails IS the diagnosis

dig fails            → DNS (topic 9)             — not a "network" problem
ping/mtr fails       → routing/host down (38,49,50)
nc refused           → service down / wrong port (39,41)
nc timeout           → firewall / security group (54,59)
curl cert error      → TLS/expiry (46)
curl 5xx             → the app / its backend (45)   — now it's an app problem

✅ Do — good use cases

Always bottom-up: DNS → ping → port → TLS → app — the first failing rung names the culprit and its team
Test the name AND the IP separately (topics 9/15) — it isolates DNS from reachability in two commands
Keep this runbook literally pinned — under incident pressure, a checklist beats intuition every time

❌ Don't — anti-patterns

Starting at the app ("must be a code bug") before confirming DNS resolves and the port is reachable — you'll debug the wrong layer for an hour (topic 1)
Stopping at the first tool you know instead of the first tool that fails — walk the layers in order

Pattern & benefit: this runbook turns the vaguest page — "it's down" — into a deterministic, five-step bisection of the whole stack, where each layer confirmed eliminates everything below it and each failure points at a specific owner (DNS team, network, firewall, TLS, or app). It's the OSI/TCP-IP model (topic 1) operationalized, and it's the single most valuable habit in this playbook.

Gotcha — why it goes wrong: the universal mistake is debugging down from assumptions instead of up from facts — engineers "know" it's the app and burn an hour there while dig would have shown DNS failing in two seconds, or nc would have revealed a firewall timeout (topic 20). Discipline beats intuition: confirm each layer before climbing. And beware ICMP being blocked (topic 15) — a failed ping isn't proof the host is down; confirm the actual port with nc.

24Latency vs bandwidth — why "fast" is two things▶

Scenario: "the connection is slow" means two totally different things — high latency (each round-trip takes long) or low bandwidth (the pipe is narrow). They have different causes and different fixes, and confusing them wastes effort.

🧠

Analogy: latency is the length of the road; bandwidth is the number of lanes. A single sports car (one small request) is limited by road length — a 200ms round-trip to another continent can't be sped up by adding lanes. A thousand trucks (a big file) are limited by lanes — widen the highway (more bandwidth) and they all arrive sooner. Adding lanes doesn't shorten the road, and shortening the road doesn't add lanes.

Code

# LATENCY — round-trip time (topic 15):
$ ping -c5 api.shop.com
  time=142 ms          ← 142ms EACH way-and-back. A cross-continent road.
# 100 sequential requests × 142ms = 14 seconds, no matter how fat the pipe.

# BANDWIDTH — throughput (how much per second):
$ iperf3 -c speedtest-host           # measure real throughput
  [SUM]  0-10s   1.10 GBytes  942 Mbits/sec    ← ~1 Gbit of lanes
$ curl -w "%{speed_download}\n" -so /dev/null https://host/big.iso

# the tell for WHICH one is your problem:
#   small requests slow, big transfers fine → LATENCY (distance, too many round-trips)
#   big transfers slow, pings fast          → BANDWIDTH (narrow pipe, congestion)

Output — the fix depends entirely on which

Problem: API feels sluggish, each call 200ms, payloads tiny
  → LATENCY. Fixes: fewer round-trips (batch calls, HTTP keep-alive topic 45,
    CDN/edge closer to users, caching). More bandwidth does NOTHING.
Problem: downloads/backups crawl, but ping is 5ms
  → BANDWIDTH. Fixes: bigger link, compression (topic 27 -z), parallel streams.

✅ Do — good use cases

Diagnose which "slow" you have: ping for latency, iperf3/big-file curl for bandwidth
Attack latency with fewer round-trips — batching, keep-alive (topic 11), caching, edge/CDN closer to users
Attack bandwidth with compression (-z, topic 27), parallelism, or a bigger link

❌ Don't — anti-patterns

Buying more bandwidth to fix a latency problem — a chatty app doing 50 sequential round-trips won't speed up with a fatter pipe
Ignoring round-trip count — N sequential requests pay the latency N times; the fix is fewer trips, not faster ones

Pattern & benefit: separating latency from bandwidth is what makes performance work targeted instead of guesswork — one is distance/round-trips (fixed by fewer trips and edge proximity), the other is pipe width (fixed by bigger links and compression). Most "slow API" complaints are latency × too many round-trips, which is why batching and keep-alive (topic 11) — not bigger servers — are the usual cure.

Gotcha — why it goes wrong: the expensive mistake is throwing bandwidth at a latency problem — a mobile app making 40 sequential API calls at 150ms each takes 6 seconds regardless of a gigabit connection, because the bottleneck is round-trip count × distance, not pipe width. The fix is architectural (batch into one call, use keep-alive to skip repeated handshakes, put a CDN closer to users), and no amount of extra bandwidth touches it. Conversely, latency-optimizing a bulk transfer that's bandwidth-bound is equally futile.

25Cloud networking mapped — VPC = your Linux knowledge▶

Scenario: everything in Parts I–II maps almost 1:1 onto AWS/cloud networking. Once you see that a VPC is just subnets, route tables, and firewalls with fancy names, cloud networking stops being a separate subject.

🧠

Analogy: a cloud VPC is your Linux networking knowledge wearing a business suit. The concepts are identical — subnets are subnets (topic 2), route tables are route tables (topic 4), firewalls are firewalls (topic 20) — AWS just gives them capitalized names, a web console, and an API. Learn the plumbing once (this playbook), and the "cloud networking" exam is mostly vocabulary translation.

The translation table

Linux / this playbook        AWS VPC equivalent
──────────────────────────────────────────────────────────
subnet + CIDR (topic 2)     VPC + Subnets (same CIDR math!)
routing table (topic 4)     Route Tables (local + 0.0.0.0/0 → IGW)
default gateway (topic 4)   Internet Gateway (IGW) / NAT Gateway
NAT / masquerade (topic 8)  NAT Gateway (private subnet egress)
iptables firewall (topic 20) Security Groups (stateful) + NACLs (stateless)
/etc/hosts, DNS (topic 9/4) Route 53 + VPC DNS (the .2 resolver)
reverse proxy / LB (47/48)   ALB (L7) / NLB (L4) — same L4-vs-L7 choice
ss listeners (topic 7)      "is the SG allowing the port?" is the new question

# the #1 cloud gotcha, in one line:
#   host firewall CLEAN + service listening + STILL timing out
#   → it's the SECURITY GROUP (a firewall OUTSIDE the host — topic 54's warning)

Output — debugging is the SAME runbook (topic 23), plus one layer

can't reach an EC2 service?  walk topic 57's layers, THEN also check:
  • Security Group     — does it allow your source IP on that port? (stateful firewall)
  • NACL               — subnet-level allow/deny (stateless — needs BOTH directions!)
  • Route table        — is there a route to IGW/NAT? (topic 4)
  • public IP / subnet  — is it even in a public subnet with a path out? (topic 8)

✅ Do — good use cases

Map cloud concepts to the fundamentals you know — VPC/subnet/route-table/SG are topics 36/38/54 renamed
Add "check the Security Group" as the extra rung on the topic 57 runbook for any cloud host
Remember SGs are stateful (reply traffic auto-allowed), NACLs are stateless (must allow both directions)

❌ Don't — anti-patterns

Treating "cloud networking" as unrelated to Linux networking — it's the same model; the plumbing knowledge transfers directly
Debugging only inside the host — the Security Group (a firewall the host can't even see) is the most common cloud "it times out" cause (topic 20)

Pattern & benefit: the fundamentals in this playbook ARE cloud networking — CIDR math (topic 2), routing (topic 4), NAT (topic 8), firewalls (topic 20), and DNS (topic 9) map directly onto VPC subnets, route tables, NAT gateways, security groups, and Route 53. The DevOps playbook's EKS networking (topics 5, 6) is this same knowledge one more layer up. Learn it once here; apply it everywhere.

Gotcha — why it goes wrong: the cloud's signature gotcha is the firewall you can't see from the host — you SSH in, iptables is empty, the service is listening (topic 7), and it still times out, because the Security Group (an external, stateful firewall — topic 54) is dropping your traffic. And NACLs trip people because they're stateless: unlike Security Groups, they don't auto-allow return traffic, so you must permit both the inbound request AND the outbound reply, or connections half-open and hang.

26Capstone — one HTTP request, every layer★ the goal▶

Scenario: the whole playbook in one journey. You type curl https://api.shop.com/orders — trace what happens through all 60 topics, from DNS to the app process and back. If you can narrate this, you're operating, not guessing.

🧠

Analogy: the full postal journey of one letter — you look up the address (DNS), the post office plots the route (routing), it's sealed in tamper-proof security envelopes (TLS), trucked across cities (TCP/IP hops), received at the mailroom (load balancer/proxy), handed to the right clerk (the app process), who reads it, pulls a file (the database), writes a reply, and the whole trip runs in reverse. Sixty topics, one round-trip.

The journey — every step is a topic you now own

$ curl https://api.shop.com/orders

1. RESOLVE    getent/dig: api.shop.com → 203.0.113.42   (topics 9,10)
              └ checks /etc/hosts, then DNS; TTL-cached (topic 9)
2. ROUTE      not my subnet → default route → gateway    (topics 2,4)
              └ NAT rewrites my private source IP on the way out (topic 8)
3. TCP        3-way handshake to 203.0.113.42:443        (topic 5)
              └ SYN → SYN-ACK → ACK  (a firewall/SG could DROP here — 54,59)
4. TLS        handshake: verify cert (name, expiry, CA)  (topic 12)
              └ agree on encryption; now the channel is private
5. HTTP       GET /orders + headers, encrypted inside TLS (topic 11)
6. PROXY      hits nginx/ALB → routes to a healthy backend (topics 13,14)
              └ health checks chose a live app server (topic 13)
7. APP        a process (topic 9) accepts the socket (fd — topic 15),
              queries the DB, builds the response
8. RETURN     200 + JSON flows back through the same layers, reversed
9. OBSERVE    every hop logged (topic 30), timed (curl -w, topic 51)

# and when ANY step fails, the topic-57 runbook tells you which:
#   step 1 fails → DNS │ 2-3 → routing/firewall │ 4 → TLS │ 6 → LB │ 7 → app

Output — the round-trip, and where each failure would surface

curl -w "dns:%{time_namelookup} tcp:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer}\n"
dns:0.004  tcp:0.031  tls:0.089  ttfb:0.140      ← each phase = a topic, timed.
# slow dns→topic43 │ slow tcp→routing/39 │ slow tls→46 │ slow ttfb→the app.
# you can now point at the exact layer for ANY symptom. That's the whole game.

✅ Do — good use cases

Narrate this journey out loud — the gaps you stumble on are exactly the topics to revisit
Map every incident to a step: which layer failed? (topic 23 is this capstone as a runbook)
Use curl -w timing (topic 17) to see the real journey's phases on any request, live

❌ Don't — anti-patterns

Treating these 60 topics as separate trivia — they're one connected system; this request touches most of them in sequence
Guessing at a failing layer when the journey (and topic 57) tells you exactly where to look

Pattern & benefit: count what one curl touches: DNS (43,44), routing and NAT (36,38,42), TCP and its firewalls (39,54,59), TLS (46), HTTP (45), proxy and load balancing (47,48), the process/fd/memory model that runs the app (9,14,15), and the logs/timing that observe it all (30,51) — narrated through the debug-in-layers discipline (57). The playbook isn't 60 facts; it's one system, and this is the tour. Master this journey and "the network is down" becomes "layer 4 is timing out, check the security group" — every time.

Gotcha — why it goes wrong: the failure modes are now all ones you can name and place: name won't resolve = DNS (43); "network unreachable" = routing (38); "connection refused" = nothing listening (39,41); "timed out" = a firewall/SG dropping (54,59); cert error = TLS (46); 502/504 = the backend crashed or stalled (45,48); 137 in the app logs = it got OOM-killed (14). The playbook doesn't stop failures — it turns every one from a mystery into a layer with a number. That is what "pro" means.

🎯 Quiz — Test Yourself

Answer 5 per round. Your progress and score are saved in this browser — come back anytime and continue where you left off.

Score: 0 / 0 answered · 0 total