Distributed SQL Systems
How Spanner and CockroachDB combine sharding, consensus, and distributed transactions to deliver global ACID over SQL.
Overview
Distributed SQL databases like Google Spanner and CockroachDB combine everything covered in this path — sharding, replication, consensus, and distributed transactions — underneath a layer that still speaks SQL and still offers real ACID transactions, even though the data is spread across many machines and regions.
Why It Exists
For a long time, the trade-off looked binary: relational databases gave you ACID and SQL but only really scaled vertically, while NoSQL systems scaled horizontally but gave up strong consistency, SQL, or both. Distributed SQL systems exist to prove that trade-off was never fundamental — by combining sharding for scale, consensus for safely coordinating replicas, and distributed transaction protocols on top, they deliver global ACID transactions over horizontally distributed data.
Real World Example
CockroachDB automatically shards a table's data into ranges, replicates each range across multiple nodes using Raft for consensus, and coordinates transactions that touch rows in different ranges — potentially in different data centers — so they still get full ACID guarantees. Google Spanner does the same at global scale, additionally using synchronized atomic clocks (TrueTimeGoogle Spanner's globally-synchronized clock, using GPS and atomic clocks to bound clock uncertainty tightly enough to order transactions globally without an extra coordination round trip.Learn more) to assign globally consistent transaction timestamps without needing an extra coordination round trip for every transaction.
Example Data
Who manages what — manually sharded MySQL vs. a distributed SQL engine
| Concern | Manually Sharded MySQL | Distributed SQL (Spanner/CockroachDB) |
|---|---|---|
| Deciding which shard a row lives on | Application code | The engine itself |
| Replication and failover | Application/ops tooling | Built-in, via Raft |
| Cross-shard transactions | Usually avoided or hand-built | Native, full ACID |
The Pieces, Combined
Sharding Underneath, SQL on Top
Data is automatically split into ranges (shards) and distributed across nodes — but the application still just writes SQL, unaware of exactly which node holds which row.
Raft for Replica Consensus
Each range is replicated across multiple nodes, with Raft handling leader election and safe log replication for that range — the same consensus mechanism covered in the previous chapter, applied per-range at scale.
Distributed Transactions Across Ranges
A transaction touching rows in multiple ranges is coordinated using a distributed transaction protocol (conceptually related to the 2PC and Saga ideas from earlier), so it still commits or aborts as one atomic unit.
Spanner's TrueTime — Global Clock Synchronization
Spanner uses GPS and atomic clocks to keep every node's clock within a tightly bounded uncertainty window, letting it assign globally meaningful transaction timestamps without a round trip to a central coordinator — a genuinely distinctive piece of engineering unique to Spanner's specific design.
Diagram
SQL in, distributed guarantees delivered underneath, transparently
Client sends SQL
Data automatically sharded into ranges
Each range replicated via Raft
Cross-range transaction coordinated
ACID guarantee delivered
despite the underlying distribution
Common Mistakes
Assuming a distributed SQL database performs identically to a single-node relational database for every query
Why: A transaction that touches multiple ranges or shards still pays real cross-node coordination cost, even though it's hidden behind familiar SQL and ACID semantics.
Fix: Design schemas and transactions to minimize cross-range operations where latency is critical, the same way you'd think about cross-shard queries in a manually-sharded system.
Choosing a distributed SQL system purely to avoid manual sharding, without actually needing its consistency guarantees
Why: These systems trade some raw throughput and operational simplicity for strong consistency guarantees manual sharding doesn't provide — if a workload would be fine with eventual consistency, a simpler horizontally-scaled NoSQL system might be a better fit.
Fix: Choose a distributed SQL system specifically because you need both horizontal scale and strong consistency/SQL together, not just to avoid manual sharding work.
Assuming every distributed SQL system's clock/timestamp mechanism works like Spanner's
Why: Spanner's TrueTime relies on specialized atomic-clock/GPS infrastructure to bound clock uncertainty tightly — a system without that infrastructure has to use a different, often costlier coordination mechanism, and assuming Spanner's exact guarantees apply elsewhere is a real, subtle mistake.
Fix: Understand the actual consistency and timestamping mechanism of the specific distributed SQL system in use, rather than assuming they all work identically.
Interview Questions
What problem do distributed SQL databases like Spanner and CockroachDB solve?
They let an application get both the horizontal scalability of a NoSQL system and the strong consistency, ACID transactions, and familiar SQL interface of a traditional relational database, instead of having to choose one or the other.
What are the main pieces, from earlier in this path, that a distributed SQL system combines to deliver ACID at scale?
Sharding (to distribute data across nodes), a consensus algorithm like Raft (to safely replicate each shard and handle leader election), and a distributed transaction protocol (to coordinate transactions that span multiple shards) — combined underneath a layer that still exposes standard SQL.
What does Google Spanner's TrueTime do, and why does it matter for global transaction ordering?
TrueTime uses GPS and atomic clock hardware across Spanner's data centers to keep every node's clock synchronized within a tightly bounded, known uncertainty window. This lets Spanner assign globally meaningful commit timestamps to transactions and know how long to wait to guarantee correct ordering, without needing an extra communication round trip to a central coordinator for every transaction — which is what makes Spanner's global consistency practical at low latency, rather than purely theoretical.
Production Best Practices
Do
✓Design schemas to minimize cross-range transactions where latency matters.
✓Choose distributed SQL specifically for workloads needing both scale and strong consistency.
✓Understand the specific consistency/timestamping mechanism of the system actually in use.
Don't
✗Don't assume every query performs like it would on a single-node database.
✗Don't adopt distributed SQL purely to avoid manual sharding if eventual consistency would be fine.
✗Don't assume Spanner's TrueTime guarantees apply to a different distributed SQL system.
Comparison
| Sharding Managed By | Cross-Shard Transactions | Consistency Guarantee | |
|---|---|---|---|
| Traditional Sharded RDBMS (manual) | Application code | Usually avoided or hand-built | Per-shard only |
| Distributed SQL (Spanner/CockroachDB) | The engine itself | Native, full ACID | Global, strong consistency |