Caching Everything: From CPU Cache Lines to Global CDNs

•Why Modern Software Spends So Much Effort Avoiding the Database

Open the network tab on almost any fast, modern application and you'll notice something odd: the database is barely in the picture. A product page loads in 40 milliseconds, but the query that originally computed that page took 200 milliseconds to run. A social feed refreshes instantly, but assembling it from raw tables would take seconds. The database didn't get faster. Something else is doing almost all of the work — a chain of caches, stacked on top of each other, each one built to avoid asking the layer below it for anything it doesn't have to.

That chain runs from the physical silicon inside a CPU all the way out to the browser sitting on a reader's laptop, and it exists at every single layer software is built on:

The Full Caching Stack — Every Layer Exists to Avoid the One Below It

CPU Registers

~0.2 ns

L1 Cache

~1 ns

L2 Cache

~4 ns

L3 Cache

~15 ns

RAM

~100 ns

Application Cache

in-process — Caffeine, Guava, Map

Redis

sub-millisecond, over the network

Database

single-digit to tens of ms

CDN Edge

tens of ms, geographically close

Browser Cache

~0 ms after the first load

“Every cache in this stack exists for exactly one reason: the layer beneath it is too slow to ask directly, every single time.”

•The Physics of Speed: A Latency Hierarchy Every Engineer Should Know

None of this is a software design preference. It's downstream of physics. An electrical signal can only travel so far in a given time, and different storage media have fundamentally different physical access characteristics — a CPU register is a handful of transistors a few millimeters from the core that needs it; a cross-region API call has to cross oceans of fiber. at each layer is a direct consequence of the distance and medium the request has to travel through, and every cache in the stack above exists to keep requests answered at the layer where the physics is cheapest.

The exact numbers below vary by hardware generation and network path, but the relative gaps between layers have stayed remarkably stable for over a decade — and they're the numbers every caching decision ultimately traces back to.

Layer	Typical Latency	Roughly How Much Slower Than L1
CPU Register	~0.2 ns	baseline
L1 Cache	~1 ns	~5x
L2 Cache	~4 ns	~20x
L3 Cache	~15 ns	~75x
RAM	~100 ns	~500x
In-process cache lookup	~100 ns – 1 µs	~500 – 5,000x
SSD (NVMe) random read	~20 – 100 µs	~100,000 – 500,000x
Redis, same data center	~0.3 – 1 ms	~1.5M – 5M x
SQL query, indexed	~1 – 10 ms	~5M – 50M x
Cross-region network round trip	~70 – 150 ms	~350M – 750M x
Third-party API call	tens to hundreds of ms	up to ~1 billion x

ns = nanoseconds · µs = microseconds (1,000 ns) · ms = milliseconds (1,000,000 ns). Figures are representative order-of-magnitude values, not a benchmark from specific hardware — see Jeff Dean's widely cited "Latency Numbers Every Programmer Should Know" for the canonical CPU/RAM figures this table is grounded in.

“Reading from an L1 CPU cache is roughly the same speed advantage over a cross-region network call as walking to your kitchen is over flying to another continent. Caching isn't an optimization technique — it's what makes the difference between those two things livable in the same request.”

•Registers, Cache Lines, and the L1 / L2 / L3 Hierarchy

The very first cache in any software system isn't Redis, or even RAM — it's built into the CPU itself, and most engineers never think about it directly because the hardware manages it automatically. Modern CPUs sit on top of a small hierarchy of increasingly larger, increasingly slower caches specifically to hide the ~100 ns cost of a full RAM access, which — at billions of instructions per second — would otherwise stall the processor constantly.

is the smallest and fastest tier, typically 32–64 KB, private to each core, and often split into a separate instruction cache and data cache. is larger (256 KB – 1 MB per core on most modern chips) and slightly slower, usually still private per core. is the largest on-chip tier — often tens of megabytes — and is normally shared across all the cores on the chip, acting as a last stop before a request has to leave the CPU package entirely and hit RAM.

Data doesn't move between these caches one byte at a time. It moves in fixed-size blocks called s — 64 bytes on almost every mainstream x86 and ARM CPU today. Fetch one byte, and the CPU actually pulls in the entire 64-byte line around it. That single implementation detail explains two of the most counter-intuitive performance facts in software engineering.

•Spatial and Temporal Locality: Why Arrays Beat Linked Lists

Because a cache line pulls in a contiguous 64-byte block, a CPU cache rewards two specific access patterns, known formally as locality of reference.

is the tendency to access memory addresses near ones you just accessed. An array stores its elements contiguously, so iterating through one sequentially means the second, third, and fourth elements are usually already sitting in the same cache line pulled in for the first — the CPU barely has to wait. A linked list, by contrast, scatters its nodes wherever the allocator happened to put them; walking it means chasing pointers across memory addresses that are almost never in the same cache line, forcing a fresh, ~100 ns RAM trip for practically every node.

is the tendency to re-access the same address again soon — a loop counter, a hot configuration value, an object you just wrote and are about to read back. Caches at every layer of the stack in this article are, at their core, bets on temporal locality: the assumption that whatever was just requested is disproportionately likely to be requested again soon.

This is the real, low-level reason "arrays are faster than linked lists" survives as engineering folklore even though both are theoretically O(n) to scan: the array's performance is dominated by cache-friendly sequential access, and the linked list's is dominated by cache-hostile pointer chasing. The algorithmic complexity is identical. The cache behavior is not.

•False Sharing and the Cache Miss Penalty

A happens when a requested address isn't present in the current cache tier, forcing a trip to the next, slower one — L1 miss falls to L2, L2 miss falls to L3, L3 miss falls all the way to RAM. Every cache-design decision in this article, at every layer, exists to keep the miss rate as low as possible, because the cost asymmetry is severe: an L1 hit costs about a nanosecond; an L1 miss that cascades all the way to RAM costs roughly 100 times that.

Multi-core CPUs introduce a subtler, uglier problem on top of this: . Cache coherence between cores is tracked at the granularity of a whole 64-byte cache line, not individual variables. If two unrelated variables — say, two different threads' counters — happen to land in the same 64-byte line, then every write by either thread invalidates that entire line for the other core, forcing it to refetch data it never actually touched. Two threads that share no logical state can still contend as if they were fighting over the same lock, purely because of memory layout. It's one of the few CPU-level performance bugs that's genuinely invisible in source code and only shows up as an unexplained multi-core scaling cliff in a profiler.

Everything from this point forward in the article — application caches, Redis, CDNs — is the same idea replayed at a larger and larger scale: keep the hot data as close as possible, and pay close attention to what actually gets evicted and when.

•In-Process Caches: Map, ConcurrentHashMap, Caffeine, and Guava

The next layer up is the first one application code actually controls directly: an in-process cache living in the same memory space as the running application, with no network hop at all — a lookup here costs roughly what a RAM access costs, not what a Redis round-trip costs.

The simplest version is a plain in-memory map. Node.js applications reach for a Map or a plain object; it's fast, has zero dependencies, and is entirely unbounded and single-instance by default — nothing evicts old entries unless the application code does it explicitly, and it holds no data other processes or instances can see.

On the JVM, the same idea needs a thread-safe implementation the moment more than one request can run concurrently, which is why shows up constantly as the bare-metal option — but it still has no eviction policy of its own, no TTL, and no size bound. For anything past a toy cache, most JVM codebases reach for a dedicated library instead: , a high-performance caching library that implements the W-TinyLFU eviction policy (covered in full in Chapter 7) and gives you size limits, TTL, and near-optimal hit rates out of the box; or its predecessor , simpler and still widely deployed in older codebases, though largely superseded by Caffeine in new projects. Spring applications typically don't call either directly — Spring Cache is an abstraction layer that can be backed by Caffeine, Redis, or several other providers, chosen with a single configuration line.

•Advantages, Disadvantages, and When Not to Use an Application Cache

The advantage is speed with zero added infrastructure: no network hop, no serialization cost, no extra service to deploy, monitor, or pay for. For read-heavy, small, slow-changing data — feature flags, parsed configuration, compiled regular expressions, a lookup table of country codes — an in-process cache is close to free performance.

The disadvantage is exactly the same property, from the other direction: the cache lives and dies with a single process. In any horizontally scaled deployment — which is most production systems today — every instance holds its own separate copy, meaning the same data gets fetched and cached N times across N instances instead of once. Worse, invalidating an entry in one instance's cache does nothing to the other N-1 copies, so a naive in-process cache used for anything that changes can silently serve stale data to some fraction of users depending on which instance they happen to land on.

The rule of thumb: reach for an in-process cache when the data is read far more than it's written, small enough to duplicate cheaply across every instance, and either effectively immutable or tolerant of a short staleness window. Reach for a distributed cache — Chapter 4 — the moment consistency across instances actually matters, or the dataset is too large to duplicate N times without a memory problem.

•Redis Architecture: Standalone, Replica, Sentinel, and Cluster

is the default answer the industry reaches for once a cache needs to be shared across every instance of an application rather than duplicated per-process — an in-memory data store living as its own service, reachable over the network, so every application instance sees the exact same cached state.

The simplest deployment is a single standalone instance — fine for development, risky in production, since one crash means total data loss unless persistence is configured (§4.2). Production deployments add a primary with one or more read replicas: writes go to the primary, which asynchronously streams changes to replicas that serve read traffic and stand ready to be promoted if the primary fails.

Promoting a replica by hand doesn't scale operationally, which is what exists for — a separate set of processes that monitor a primary/replica group, detect failure, and automatically promote a replica, without changing anything about how the data itself is stored or sharded.

Neither replicas nor Sentinel solve a different problem: what happens when the dataset is too large for one machine's RAM, or write throughput exceeds what a single primary can handle? That's what is for — Redis's native horizontal-scaling mode. Every key is mapped, via CRC16, into one of 16,384 fixed s, and each slot is owned by exactly one primary node (each of which can still have its own replicas for availability). Nodes gossip cluster state directly with each other rather than relying on a separate coordinator. The one sharp edge worth knowing up front: a multi-key command like MGET key1 key2 fails with a CROSSSLOT error if the keys land on different nodes — Redis Cluster deliberately refuses silent cross-node multi-key operations rather than serializing them invisibly. Applications that need related keys to always co-locate use hash tags (user:{123}:profile, user:{123}:sessions — only the {123} portion is hashed) to force them into the same slot on purpose.

•Persistence: RDB Snapshots and the Append-Only File

Redis is fundamentally an in-memory store, which raises an obvious question: what happens on restart? Redis offers two independent, combinable persistence mechanisms, and understanding both matters because they trade durability against performance in opposite directions.

takes a point-in-time binary snapshot of the entire dataset at a configured interval — compact, fast to load on restart, but anything written since the last snapshot is gone if the process crashes. instead logs every write command as it happens, replaying the whole log on restart to rebuild state — far more durable (configurable down to fsync on every single write), but a larger file and a slower restart than loading one compact RDB snapshot.

Most production deployments run both together: RDB for fast, compact backups and quick recovery, AOF for the durability guarantee between snapshots. Either way, this remains an important mental model to hold onto for the rest of this article: Redis is a cache with an optional durability layer bolted on, not a database with caching bolted on — treat data that only lives in Redis, with persistence disabled, as genuinely disposable.

•Redis Data Structures: Beyond Simple Key-Value

What separates Redis from a plain key-value store like Memcached is that values aren't just opaque bytes — Redis exposes a small set of real data structures, each with its own atomic operations, and reaching for the right one is often the difference between a clean one-line solution and hundreds of lines of application-level bookkeeping.

String — the simplest type: bytes, a number, a serialized object. INCR/DECRBY are atomic, which is why raw counters (page views, rate-limit counts) almost always use a plain String.
Hash — a field-value map inside a single key, ideal for representing an object (a user record: name, email, plan) without the overhead of separate keys per field, and lets you fetch or update one field without touching the rest.
List — an ordered, linked sequence supporting fast push/pop from either end — the natural structure for queues and, in reverse, a capped "recent activity" feed.
Set — an unordered collection of unique members with fast membership tests (SISMEMBER) and set algebra (SINTER, SUNION, SDIFF) — useful for tags, unique visitor tracking, and "mutual friends"-style intersection queries.
— a set where every member also carries a floating-point score, kept in sorted order automatically. This single structure is the backbone of every real-time leaderboard you've ever used (§4.4).
— not a distinct type but a way of addressing a String at the bit level, for compact boolean flags at massive scale — a 100-million-user daily-active-users bitmap fits in about 12.5 MB.
— a probabilistic structure that estimates the cardinality (count of distinct elements) of a set using a fixed ~12 KB of memory regardless of whether it's tracking a thousand or a billion distinct items, at roughly 0.81% standard error — the trade you make when you need "how many unique visitors" and genuinely don't need the exact number.
— an append-only log structure with consumer-group semantics, closer to a lightweight Kafka than a cache primitive, and the structure event-driven invalidation (§9.3) leans on directly.

•Real Use Cases: Leaderboards, Sessions, Rate Limiting, Carts, and OTPs

These aren't hypothetical — they're the handful of problems Redis gets reached for in almost every production system that has it, and each one maps to a specific data structure rather than a generic cache-aside pattern.

Leaderboards — ZADD a player's score into a Sorted Set, ZREVRANGE for the top N, ZRANK for any single player's live rank. All O(log N), all correct under concurrent updates, with none of the application-level sorting a naive SQL ORDER BY score DESC LIMIT 10 under heavy write load would require re-running constantly.
Session storage — a Hash per session ID, with a TTL matching the session's expected lifetime, is why logging out on one device instantly reflects on another: the session state lives in one shared place, not duplicated per app-server process (§3.2's exact problem, solved).
Rate limiting — INCR a per-user, per-window counter key with a TTL equal to the window length is the simplest correct fixed-window limiter; sliding-window variants use a Sorted Set keyed by timestamp and ZREMRANGEBYSCORE to expire old entries — both lean on Redis's atomic operations to stay correct under concurrent requests without a database round-trip on every single API call.
Shopping carts — a Hash per cart (product ID → quantity), TTL'd to expire abandoned carts automatically, avoiding a database write on every single add-to-cart click while checkout still reads and writes through to the database as the durable source of truth.
OTPs (one-time passwords) — a String with a short TTL (typically 60–300 seconds) is close to the ideal use case for Redis: naturally expiring, security-sensitive enough that letting it live in the primary database's durable storage past its usefulness is a genuine liability, not just an efficiency question.

•Browser Cache, Proxy Cache, and the Reverse Proxy Layer

Everything up to this point has been caching inside a system's own infrastructure. HTTP caching moves the conversation outward — to caches that sit between a server and the people actually making requests, some of which the server doesn't control at all.

The browser cache is the closest one to the user: a local store on the visitor's own device that can serve a repeat request without a network call being made at all, governed entirely by the response headers a server sent the first time (§5.2). A proxy cache — whether a corporate forward proxy or an ISP-level one — sits somewhere on the path between many clients and the origin, and can serve a shared cached response to different users who happen to request the same URL, which is powerful and also exactly why caching anything containing per-user data at this layer is a real privacy bug waiting to happen.

A sits in front of the origin server rather than in front of clients — is the canonical purpose-built HTTP caching reverse proxy, still widely deployed directly in front of origins specifically for its cache-hit speed and fine-grained VCL configuration language, alongside general-purpose reverse proxies like NGINX that can cache too, just with less specialized tooling for it.

•The Headers That Actually Control Caching: Cache-Control, ETag, Expires, Last-Modified

Every HTTP cache in this chapter — browser, proxy, reverse proxy, CDN — is governed by the same small set of response headers, and getting them wrong is one of the single highest-leverage mistakes in web performance, in both directions: too aggressive and users see stale content; too conservative and a static asset gets re-fetched on every single page load for no reason.

Cache-Control — the modern, primary directive. max-age set in seconds controls freshness lifetime; no-cache means "you may cache this, but revalidate with the server before serving it" (a common source of confusion — no-cache does not mean "don't cache"); no-store means genuinely never cache this, anywhere; private restricts caching to the end-user's own browser, never a shared proxy or CDN — the correct directive for anything containing per-user data.
ETag — an opaque fingerprint (often a content hash) of a specific response version. On a repeat request, the client sends it back via If-None-Match; if it still matches, the server can skip re-sending the full body entirely.
Last-Modified — a timestamp-based, coarser alternative to ETag, paired with If-Modified-Since on the follow-up request. Less precise than a content hash, but cheaper to compute for servers that can't easily fingerprint every response.
Expires — the older, pre-Cache-Control mechanism, an absolute date rather than a relative age. Still respected by clients that don't understand Cache-Control, but max-age takes precedence wherever both are present in a modern stack.

http

GET /product/42 HTTP/1.1
If-None-Match: "a1b2c3-etag-value"

--- server still has the same version ---

HTTP/1.1 304 Not Modified
Cache-Control: max-age=3600
ETag: "a1b2c3-etag-value"
(no response body sent at all)

“A 304 Not Modified is the cheapest possible successful HTTP response — the server confirms nothing changed, and the client keeps serving what it already has, with zero body bytes sent over the wire.”

•Where Varnish, Cloudflare, Fastly, and Akamai Actually Fit

It's easy to lump these four together as "HTTP caching tools," but they sit at genuinely different points in the request path and solve different problems. Varnish is self-hosted software you deploy and operate yourself, typically directly in front of a specific origin, giving full control over caching logic at the cost of running and scaling it yourself. Cloudflare, Fastly, and Akamai are the CDN layer proper — a globally distributed network of edge servers you don't operate, caching your content close to every visitor worldwide rather than at one location near your origin. Chapter 6 is dedicated entirely to how that CDN layer actually works internally, because it deserves its own treatment rather than being folded into general HTTP caching.

•Origin, Edge, and the Anatomy of a Cache Hit

A CDN's entire value proposition rests on one architectural move: instead of every request traveling all the way to a single origin server, requests hit a nearby edge server first — one of many geographically distributed points of presence a CDN operates — and only travel to the origin when the edge doesn't already have a cached copy.

The request path: a visitor's DNS resolution routes them to the nearest edge (often via , the same networking technique that lets one IP address resolve to whichever physical location is closest to the requester). The edge checks its local cache. If the content is present and still fresh (a cache hit), it's served directly from the edge — the origin server never even sees the request. If it isn't (a cache miss), the edge fetches it from the origin, serves it to the visitor, and stores a copy for the next visitor who asks for the same thing.

•Warm vs. Cold Cache, Regional POPs, and Purge

A cache is cold immediately after a purge, a deploy, or simply for content nobody nearby has requested recently — every request in that state is a miss, each one paying the full origin round-trip. A warm cache is one that's already serving hits for the content in question; high-traffic sites effectively never see a fully cold cache for their popular pages, because enough concurrent global traffic keeps re-warming it continuously.

Providers differ meaningfully in how many of these edge locations, or POPs, they operate — and it's a genuine architectural trade-off, not just a marketing number. Akamai's network is the largest by location count, with edge servers embedded directly inside individual ISP networks worldwide. Cloudflare operates a large, broadly distributed network reaching several hundred cities globally. Fastly runs deliberately fewer, more heavily provisioned points of presence — the bet being that concentrated capacity at fewer locations, plus a technique called tiered caching, gets most of the latency benefit without the operational complexity of thousands of tiny edges.

Tiered caching (Cloudflare, Fastly, and Akamai all offer versions of this) inserts a middle tier between edge and origin: if a nearby edge doesn't have the content, it asks a designated upper-tier data center before ever going all the way to origin. This means far fewer of a CDN's total locations ever hit the origin directly, which both reduces origin load and — because a piece of content only needs to warm the smaller set of upper-tier caches, not every single edge independently — improves the effective cache hit ratio globally.

Purge is how stale content gets forcibly evicted before its TTL naturally expires — a hard requirement the moment content changes and can't wait for natural expiry (a price update, a corrected article, a deleted image). Most CDNs support purging by exact URL, and the more useful version for real applications: purging by tag or surrogate key, so one API call can invalidate every cached response that includes a specific product, author, or category, without the application needing to enumerate every individual URL that might reference it.

•FIFO, LRU, LFU, and Random Eviction

Every cache in this article so far has an unstated constraint: it's finite. CPU cache is measured in megabytes, RAM in gigabytes, even Redis and CDN edges have real limits. The moment a cache is full and a new entry needs to go in, something existing has to be evicted — and which eviction policy a cache uses has a direct, measurable effect on hit rate under real-world access patterns.

FIFO (First In, First Out) — evicts whatever was inserted longest ago, regardless of how recently or frequently it's been accessed since. Simple to implement, and almost always the worst-performing policy for real workloads, because insertion order has no correlation with actual usefulness.
(Least Recently Used) — evicts whatever hasn't been accessed for the longest time, not inserted longest ago. Dramatically better than FIFO for most workloads, because it directly exploits temporal locality (§2.2) — the industry's default eviction policy for decades, and still the right choice for a large share of caches.
(Least Frequently Used) — evicts whatever has been accessed the fewest total times. Handles a specific failure mode LRU is weak on: a burst of one-time scans (a backup job reading every row once) can flush an LRU cache of genuinely hot data that just hadn't been touched in the last few seconds. LFU's own weak spot is the mirror image — an item that was extremely popular last week but has gone cold can sit in the cache indefinitely on old frequency count alone, unless the implementation ages counts down over time.
Random eviction — exactly what it sounds like: pick an entry at random and evict it. Counter-intuitively not the worst option in practice, because it has zero bookkeeping overhead and, unlike LRU, has no worst-case access pattern that defeats it entirely (a specific access sequence can be constructed to make LRU thrash; random eviction has no such adversarial case).

•ARC and (W-)TinyLFU: When Simple Isn't Enough

Plain LRU and LFU both have known failure modes, which is what led to two more sophisticated eviction algorithms that dominate serious production caches today.

(Adaptive Replacement Cache), developed at IBM's Almaden Research Center, tracks two lists — recently-used entries and frequently-used entries — plus "ghost" lists recording recently evicted keys from each, using hits against those ghost lists to continuously and automatically rebalance how much cache space favors recency versus frequency, without any manual tuning. It's shipped in real, serious infrastructure: IBM's own DS6000/DS8000 storage systems, and ZFS's adaptive replacement cache is directly descended from it.

(and its refined successor, Window TinyLFU) takes a different angle: rather than deciding what to evict, it decides what to admit in the first place. A compact frequency sketch (a 4-bit Count-Min Sketch, in Caffeine's implementation) estimates how often a candidate key has historically been requested; a new entry only displaces an existing one if it's estimated to be more valuable, and a small "admission window" gives brand-new entries a fair chance to prove themselves popular before that frequency comparison kicks in. This is the exact algorithm behind (§3.1), and it's not a marginal improvement — published benchmarks show W-TinyLFU reaching around 99% of the theoretical-optimal hit rate (Bélády's algorithm, which requires impossible-in-practice knowledge of the future) while still running in O(1) time per operation.

•Cache-Aside and Read-Through

Eviction policy decides what leaves a cache. Write strategy decides how data enters it and stays synchronized with the database in the first place — and this is where a genuinely large share of production caching bugs originate, because getting a write strategy subtly wrong doesn't fail loudly; it just serves stale or inconsistent data intermittently.

(also called lazy loading) — the application checks the cache first; on a miss, it reads the database itself and writes the result into the cache before returning it. The most common pattern in practice, precisely because it puts the application fully in control and works with any cache and any database with no special integration.
Read-Through — structurally similar to cache-aside from the outside, but the cache itself (via a provider or library, not application code) is responsible for loading a missing value from the database on a miss. The application only ever talks to the cache. Cleaner separation of concerns, at the cost of needing a cache implementation that actually supports this integration (Amazon DAX, §10.4, is a real example).

•Write-Through, Write-Around, and Write-Back

These three describe what happens on a write, not a read, and each makes a different trade-off between consistency, latency, and data-loss risk.

— every write goes to the cache first, which synchronously writes it through to the database before the write is considered complete. Cache and database are never out of sync, at the cost of every write paying the database's full latency anyway — the cache adds safety, not write speed.
Write-Around — writes go directly to the database, skipping the cache entirely; the cache only gets populated the next time that data is read. Avoids filling the cache with data that's written once and rarely re-read (a common access pattern for write-heavy logs or audit tables), at the cost of the first read after any write always being a guaranteed cache miss.
(also called write-back, matching the same term used for CPU cache write policy in §2 — same underlying idea, replayed at the application layer) — writes land in the cache and are acknowledged as complete immediately; the actual database write happens asynchronously afterward. The fastest of the three for write latency, and the riskiest: if the cache crashes before the deferred write reaches the database, that data is gone. Real systems that use this (Amazon DAX among them) pair it with replication across multiple cache nodes specifically to bring that risk down to an acceptable level, not eliminate it.

•Refresh-Ahead: Beating the Expiry, Not Reacting to It

is the odd one out — a strategy for reads, but proactive rather than reactive. Instead of waiting for a cache entry to expire and serving a slow miss to whichever unlucky request arrives first, the system refreshes a hot entry from the database shortly before its TTL runs out, based on its access pattern (Caffeine and several managed caching services support this natively). The trade is real infrastructure work — background refresh jobs, and wasted database reads for entries that turn out to see no more traffic before their original expiry — in exchange for popular keys effectively never producing a slow cache miss for an end user at all.

•Why Phil Karlton Was Right

"There are only two hard things in Computer Science: cache invalidation and naming things." The line is attributed to Phil Karlton, a Netscape engineer, and the earliest documented trace of it online is Tim Bray recalling hearing it around 1996–97 — decades old, and it hasn't aged a day, because the underlying problem hasn't gotten any easier: a cache is, definitionally, a second copy of the truth, and the moment the original changes, the system has to somehow know to update or discard every copy of that stale answer, everywhere it might be sitting — in an L3 cache, an application process, a Redis cluster, and dozens of CDN edges simultaneously.

Every technique in the rest of this chapter is a different answer to exactly one question: how does a cache find out that what it's holding is no longer true?

•TTL, Versioning, and Explicit Delete

The three simplest invalidation strategies, roughly in order of how much they trust time versus how much they trust an explicit signal:

(Time to Live) — the cache entry simply expires after a fixed duration, whether or not the underlying data actually changed. The simplest strategy to reason about and the most widely used by a huge margin, but it's a blunt instrument: too short and the cache barely helps; too long and stale data survives for exactly as long as the TTL says, no matter how wrong it's become.
Versioning — instead of invalidating a key, change its identity. A cache key that embeds a version number or content hash (product:42:v7) never goes stale in place — a new version is simply a new key, and old versions age out naturally via normal eviction rather than needing to be actively found and deleted. This is exactly the mechanism behind cache-busting static asset filenames (app.a1b2c3.js) that browsers and CDNs alike can cache with an effectively infinite TTL, safely, because a content change always produces a new filename.
Explicit delete — the application, on a write, directly deletes (or updates) the specific cache key affected. The most precise option and the one most exposed to bugs: it requires the write path to correctly know and enumerate every cache entry that might be affected by that specific change, which gets genuinely hard the moment one piece of underlying data feeds multiple different cached views.

•Event-Driven Invalidation: Pub/Sub, Streams, CQRS, and CDC

TTL, versioning, and explicit delete all assume the same process that wrote the data also knows exactly what to invalidate. That assumption breaks down completely in a microservices architecture, where the service that owns a piece of data and the services caching derived views of it are, deliberately, not the same codebase — which is what pushes invalidation from an application-level concern into an architectural one.

— a service publishes an "this changed" event to a channel the moment it writes; every other service caching that data subscribes and invalidates its own local copy on receipt. Simple and low-latency, with one real gap: Redis Pub/Sub is fire-and-forget — a subscriber that's briefly disconnected simply misses the message, with nothing replayed later.
— the more durable version of the same idea: events are appended to a persistent log rather than broadcast and forgotten, so a consumer that was offline can catch up on everything it missed once it reconnects, at the cost of slightly more operational complexity than plain Pub/Sub.
CQRS invalidation — in a Command Query Responsibility Segregation architecture, the write side already emits a structured event for every state change as a core part of the pattern, not a bolt-on; the read side's caches subscribe to that same event stream as their natural, already-existing invalidation signal, rather than needing a separate mechanism built specifically for cache invalidation.
Database triggers — a trigger fires on row change and pushes an invalidation event out directly from the database layer. Guarantees no write can bypass invalidation (unlike relying on application code to remember to invalidate), at the cost of business logic leaking into the database layer and a genuinely under-appreciated risk: it reintroduces exactly the one layer down, since firing the trigger and committing the row change aren't a single atomic guarantee.
(Change Data Capture) — instead of triggers or application code publishing events, CDC tools (Debezium is the standard) tail the database's own write-ahead log directly and turn every committed row change into a stream of events. This is the most architecturally clean answer available today, because invalidation becomes a strict, guaranteed side effect of a write actually being durably committed — not a second, separate step that can fail, be forgotten, or race with the write itself, which is precisely the class of bug the next chapter's mistakes are mostly about.

•Netflix: EVCache and Caching as a Hidden Microservice

Netflix runs one of the most heavily documented large-scale caching layers in the industry, called EVCache — a distributed, memcached-based caching system built specifically for Netflix's AWS footprint, with data globally replicated so a request in any region can be served from a nearby cache rather than crossing regions to reach a single source of truth.

The scale is genuinely hard to picture from outside: per Netflix's own engineering blog, EVCache has run across roughly 18,000 server instances holding on the order of 14 petabytes of cached data — for a single caching layer, not the whole company's infrastructure. Netflix's own engineers have publicly described caching internally as "the hidden microservice": every other service depends on it so completely, and so invisibly, that it functions as critical infrastructure without most engineers ever interacting with it directly.

•Meta: Memcache at Facebook Scale, and Why TAO Exists

Facebook's 2013 paper, "Scaling Memcache at Facebook," remains one of the most-cited real-world caching case studies in systems engineering, and for good reason: a single user loading one page could fan out into hundreds of individual memcache lookups, served by a cluster of hundreds of memcached instances, with keys distributed across them via consistent hashing.

That same paper is also the origin story for one of the most instructive lessons in this entire article: plain lookaside caching (§8.1) eventually stopped being enough for Facebook's specific access pattern — the social graph, where the same underlying data (a friendship, a like, a comment) needs to be read through dozens of different query shapes across the product. Facebook's answer was TAO, a purpose-built, graph-aware caching layer sitting in front of MySQL, understanding objects and the associations between them natively rather than caching raw, opaque query results the way a generic lookaside cache does. The published numbers are extraordinary for a cache layer: TAO has been reported sustaining on the order of a billion reads per second. The lesson generalizes well beyond Facebook's specific scale: a generic cache-aside pattern works remarkably far up the scale curve, right up until an access pattern is specific and demanding enough that a purpose-built caching layer, aware of your actual data shape, stops being optional.

•Cloudflare: Tiered Caching at the Edge

Cloudflare's Tiered Cache is worth calling out specifically because it's the clearest publicly documented example of the warm/cold and origin/edge concepts from Chapter 6 implemented at real, internet-wide scale. Rather than treating every one of its edge locations as equally capable of hitting the origin, Cloudflare organizes them into a hierarchy: a request that misses at a nearby, lower-tier edge doesn't go straight to origin — it first asks a designated upper-tier data center. Only if that upper tier also misses does a request finally reach the origin server at all.

The direct effect is that origin servers see a small, concentrated number of connections instead of one from every edge location worldwide independently — real, measurable bandwidth and origin-load savings purely from restructuring which caches are allowed to talk to origin directly, without changing a single line of application code.

•Amazon: DAX and ElastiCache — Managed Caching as a Product

Amazon offers two distinct managed caching products worth telling apart, because the difference itself is a useful lesson in cache specialization. ElastiCache is a general-purpose managed Redis or Memcached service — you get the exact same Redis this chapter has covered throughout, just operated by AWS instead of self-hosted, usable in front of any data source.

DAX (DynamoDB Accelerator) is narrower and more specialized by design: a caching layer purpose-built exclusively for DynamoDB, sitting transparently in front of it so existing application code barely has to change. Its official positioning is blunt about the payoff: up to a 10x performance improvement, taking typical read latency from milliseconds down to microseconds. Architecturally, DAX runs as a cluster of nodes and internally combines write-through and write-behind (§8.2) rather than committing to just one — writes update the DAX cache and DynamoDB together, giving read-heavy DynamoDB workloads a caching layer without the application needing to hand-implement cache-aside logic itself. It's a concrete illustration of a pattern this article keeps returning to: a cache purpose-built for one specific data source, deeply, tends to outperform a generic one bolted on from the outside.

•A Decision Framework: Matching Data to the Right Cache Layer

Every concept in this article compresses down to one recurring, practical question: given a specific piece of data, where should it actually live? There's no universal answer, but there is a reliable framework — access frequency, size, consistency requirements, and blast radius if it's wrong.

Data	Where to Cache	Why
User session	Redis	Shared across every app instance; must survive a single instance restarting
Product catalog	Redis + CDN	Read far more than written; safe to serve slightly stale to most visitors
Static images / JS / CSS	CDN + Browser Cache	Immutable once built (with content-hashed filenames) — cache aggressively, for a very long time
Feature flags / config	Application Cache	Small, read on every request, safe with a short staleness window
Frequently used lookup tables	Application Cache or Redis	Depends purely on whether it must be identical across every instance
Real-time leaderboard	Redis (Sorted Set)	Needs atomic, concurrent-safe ranked updates — a database ORDER BY doesn't scale to this write pattern
OTP / short-lived token	Redis	Naturally expiring; doesn't belong in durable storage past its usefulness window
Financial transaction record	Don't cache (or cache read-only, TTL near-zero)	Correctness matters more than speed; caching this class of data is a liability, not an optimization
Analytics / raw event log	Don't cache	Write-once, rarely re-read in the same shape twice — a cache here mostly just wastes memory

•Caching Everything, Infinite TTLs, and Stale Data

The single most common caching mistake isn't a subtle algorithmic one — it's reaching for a cache reflexively, for data that didn't need one. Caching read-once data wastes memory for zero benefit. Caching write-heavy data adds invalidation complexity (§9) without a real read-speed payoff, since it barely stays valid long enough to be read from the cache before changing again. And an infinite or excessively long TTL, set once under time pressure and never revisited, is how "the cache" quietly becomes the place a team debugs first every time data looks wrong in production — a strong signal the caching strategy has stopped being trusted, which defeats the entire point of having one.

•Stampede, Avalanche, Penetration, and the Dogpile Effect

Three closely related failure modes get confused with each other constantly, and they call for genuinely different fixes, so the distinction is worth being precise about.

(also called a cache stampede, or the dogpile effect) — one single hot key expires, and every one of the requests that were relying on it arrive at the database simultaneously, all trying to recompute the exact same value at once. The standard fix is a locking or "single-flight" pattern: the first request to miss recomputes the value while every other concurrent request for that same key waits on the result, instead of each one independently hammering the database.
— the same failure, at a larger and more dangerous blast radius: not one key, but many keys expiring at close to the same moment (a common, easy-to-cause-by-accident bug: setting the exact same fixed TTL on a large batch of keys warmed at the same time), producing a synchronized wave of database load across an entire dataset rather than one hot key. The standard fix is jittered TTLs — adding a small random offset to each expiry so keys spread their expiration out over time instead of firing in lockstep.
— requests for a key that doesn't exist anywhere — not in the cache, and not in the database either — which means the cache offers zero protection, since there's genuinely nothing to cache, and every single such request falls all the way through to the database regardless of traffic volume. This is also a real, exploitable attack surface: an attacker can deliberately request a flood of non-existent IDs specifically to bypass the caching layer entirely. The standard fix is to explicitly cache the negative result too (a short-TTL "this doesn't exist" marker) or use a Bloom filter in front of the cache to reject known-nonexistent keys before they ever reach the database.

•Memory Leaks and Unbounded Caches

An application cache with no size limit (§3.1's plain Map is the classic example) is a memory leak with a delay timer on it — harmless in development, and a slow-motion out-of-memory crash in production once traffic and unique keys grow past whatever the box's RAM was sized for. Every serious in-process caching library (Caffeine, Guava) enforces an explicit maximum size or weight as a first-class, mandatory setting for exactly this reason — treat an unbounded cache as a bug, not a simplification, the same way an unbounded queue or an unclosed connection would be.

“A cache with no eviction policy and no size limit isn't a cache. It's just a second, slower-growing copy of your data, quietly waiting to run the process out of memory.”

•Emerging Trends: Edge Computing, AI-Driven Caching, and the Future

Everything in this article is well-established, production-proven engineering. What's actually shifting is where the logic deciding what to cache runs, and how much of it stops being a fixed policy an engineer configures once.

Edge computing is pushing real application logic — not just static cached responses — out to the same CDN edge locations covered in Chapter 6, running actual code close to users rather than only serving pre-cached content from them. Serverless caching platforms are emerging specifically to fit that model: ephemeral, per-region, spun up alongside the compute rather than run as a separately provisioned, always-on service the way Redis traditionally is. Persistent memory technology — non-volatile RAM that survives a restart — is quietly eroding one of this article's oldest and most foundational assumptions: that fast necessarily means volatile, and that anything at RAM speed has to be rebuilt from scratch after a crash.

The most speculative direction, and the one worth watching rather than betting production infrastructure on today, is AI-assisted caching — using access-pattern prediction to eagerly warm the specific cache entries a model expects to be needed next, before they're actually requested, rather than relying purely on TTL and reactive eviction policies designed decades before workloads looked anything like this. Whether that outperforms a well-tuned W-TinyLFU (§7.2) in practice, broadly, at general-purpose scale, is genuinely still an open, actively researched question — not yet a settled industry consensus the way write-through versus write-behind trade-offs already are.

•Conclusion: The Cache Is the System

Strip away the specific technologies in this article — CPU cache lines, Caffeine, Redis, Varnish, Cloudflare — and one idea survives every single layer: performance at scale isn't primarily a story about doing work faster. It's a story about not doing the same work twice, and about how honestly a system admits when what it's holding has gone stale.

That's also why Phil Karlton's line has outlived every specific technology named in this piece by decades. The tools for storing a value close to where it's needed have gotten remarkably good — from a 64-byte cache line to a globally distributed CDN edge, all of it is, today, mostly solved engineering. Knowing the instant that value stops being true, and reliably telling every copy of it, everywhere it's sitting, is the part that's never gotten any easier. Every architecture in Chapter 10 — Netflix's EVCache, Meta's TAO, Cloudflare's tiered edge, Amazon's DAX — is, underneath its specific implementation, still just a different, hard-won answer to that exact same question.

“Caching is not an optimization you add at the end. It's a correctness problem wearing a performance problem's clothes — and the moment a system forgets that, it starts serving confidently wrong answers, fast.”

Why Caching Exists — The Physics of Speed