Network, Internet and Browser

2.Four Minutes to Node 2

A

In this chapter

Anna follows a fan's request from the outside in — browser, DNS, internet, CDN, load balancer — and finds the cause in four minutes: an expired HTTPS certificate on one load balancer node. We'll review how the network path works (Acts 11, 12, 19), explained as following a letter from the sender's door to the right desk.

12–14 min

The Problem in Real Life

10:01:40. Anna turns off Wi-Fi on her phone, opens blueticket.example on mobile data, and reloads five times. Three times, the site loads. Twice, the browser shows a full-page warning: "Your connection is not private."

"Half the time, the browser doesn't trust us," she says. "Not slow. Not down. Not trusted." John's fingers stop moving. Everyone in the room who's done Act 19 thinks the same word at the same time: certificate.

J

Half the time means one of several machines. Find which one.

John

"The Site Is Broken" vs. Following the Request Hop by Hop

Many hops before the app

Browser, DNS, internet, CDN and load balancer all sit between a fan and the code.

Trust can fail too

If HTTPS fails, the browser refuses to continue — the request never reaches the app.

"Sometimes" problems

An error that happens half the time usually means one of several copies is different.

Network, Internet and Browser: The Path to the Servers

The letter analogy: a letter goes from the sender's house to a post box, to a sorting centre that looks up the address, across the country's roads, to the right building, where a receptionist checks the visitor's ID and sends it to the right desk. If letters arrive only half the time, you follow one letter, stop by stop, until you find the stop where it disappears.

  • The browser — the sender (Act 12): the fan's browser builds an HTTPS request, checks the server's certificate, renders the page. Her phone's browser gave the most important clue: "not private" means the TLS check failed (Act 22). DevTools (Act 17) would show the same as NET::ERR_CERT_DATE_INVALID.
  • DNS — the sorting centre's address book (Acts 12, 19): she runs dig blueticket.example: it returns the load balancer's addresses, as expected. DNS is fine.
  • The internet — the roads (Act 11): packets travel through ISPs and routers. Fans on different networks and in different countries see the same half-and-half pattern, so it isn't one ISP. The internet is fine.
  • The CDN — the local branch office (Act 12): images and the seat-map files come from the CDN, and they load every time. The CDN is fine — the failures are only on requests that go to the load balancer.
  • The load balancer — the receptionist (Acts 14, 19): it spreads requests across nodes, and it's where HTTPS is handled: it shows the certificate. After the Sale Day capacity work, there are two load balancer nodes. Half the time... two nodes.
Table — Each hop, the Act that taught it, and the result
HopLetter versionTaught inSale Day result
BrowserThe senderActs 12, 17"Not private" half the time
DNSAddress bookActs 12, 19Correct ✓
InternetThe roadsAct 11Same everywhere ✓
CDNLocal branchAct 12Serving ✓
Load balancer + TLSReceptionist checking IDActs 14, 19, 22node-2 certificate expired ✗

Following one request from the outside in

Browser

"connection not private" — first clue

lookup

DNS

dig: correct addresses ✓

packets

Internet

same pattern on every network ✓

CDN

images and seat map load ✓

TLS handshake

Load balancer node-1

certificate valid ✓

Load balancer node-2

certificate EXPIRED 00:00 ✗

Asking each node for its certificate
dig +short blueticket.example
# 203.0.113.10 (node-1)
# 203.0.113.11 (node-2)
openssl s_client -connect 203.0.113.10:443 -servername blueticket.example </dev/null 2>/dev/null \
| openssl x509 -noout -enddate
# notAfter=Mar 14 23:59:59 2027 GMT <- node-1: valid
openssl s_client -connect 203.0.113.11:443 -servername blueticket.example </dev/null 2>/dev/null \
| openssl x509 -noout -enddate
# notAfter=Dec 12 00:00:00 2026 GMT <- node-2: expired at midnight

10:04 — found: Anna asks each node for its certificate directly. node-1: valid until next March. node-2: expired at 00:00 today. Four minutes after the first error, she says: "Node 2. Expired certificate. Take it out." John removes node-2 from the DNS and load balancer pool. At 10:05, the error rate falls to almost zero. Node-1 and autoscaling (chapter four) carry the load. Samantha breathes out for what seems like the first time since 9:58.

Why didn't the alarm fire? In Act 19, Anna set up auto-renewing certificates and an alert 14 days before any certificate expires. But node-2 was added on Thursday by hand, by a script someone ran from a laptop, which copied an old certificate file instead of using the managed one. So node-2 was never part of the auto-renewal — or the alert. Acts 19 to 21 had said it again and again: anything done by hand escapes the automation. The blameless review on Monday adds one action: load balancer nodes are created only through Infrastructure as Code (Act 20), with the managed certificate.

Key Takeaway

Follow a failing request from the outside in, hop by hop: browser (the first clue — here, "connection not private" meant TLS failed), DNS, the internet, the CDN, the load balancer. An error that happens half the time usually means one of several copies is different. Here, one hand-added load balancer node carried an old, expired certificate outside the auto-renewal and its alert — found in four minutes, fixed by removing the node, prevented by creating nodes only through code.

Why This Matters

Many real outages live on the path to the servers — DNS records, CDNs, load balancers, certificates — not in the code. Knowing every hop, and how to test each one directly, lets you find them in minutes instead of hours. And "half the time" is one of the most useful clues in all of debugging.

10:05. Fans are flowing through. Now the rest of the system faces the biggest wave it has ever seen — and Anna watches, one by one, the parts she helped build all year doing their jobs.

Next