Splitting rows across databases
Horizontal sharding splits the rows of a table across several databases, each holding a different subset. Every shard has the same schema, and none of them has all the data.
The distinction that matters
A replica holds everything. A shard holds a slice.
| Replica | Shard | |
|---|---|---|
| Holds | All the data | Some of the rows |
| Solves | Too many reads | Too much data, too many writes |
| Losing one | Costs capacity | Loses those rows |
| Schema | Identical, by replication | Identical, by your migrations |
Read that third row twice. A replica going down is a capacity problem. A shard going down is an availability problem for the users whose rows live on it. That difference changes what "the database is up" means: it stops being a yes-or-no question.
flowchart TB
R["replica"] --> R1["a full copy"]
R1 --> R2["lose it: less read capacity"]
S["shard"] --> S1["one slice of the rows"]
S1 --> S2["lose it: those users are down"]
S2 --> S3["'is the database up?' stops being yes or no"]
What it is for
It solves two problems, and only two.
Write throughput. A replica can't help with writes, because every write still goes to the one primary. Short of multi-primary replication, which brings conflict problems of its own, sharding is the usual way to have more than one machine accepting writes for the same table.
Data volume. When a table no longer fits comfortably on one machine, where "comfortably" means indexes in memory, vacuum finishing and backups completing within a window, splitting the rows is the answer.
If your problem is neither of those, sharding is the wrong tool, and an unusually expensive one, for the reasons the next lessons describe.
The vocabulary
A shard is one of the databases holding a subset.
The shard key is the column whose value decides which shard a row lives on. Choosing it is the whole design, and lesson 2 explains why.
Fan-out is querying every shard and merging the results in your application, which any question that one shard can't answer requires.
Rails knows about this
Sharding is a first-class Active Record concept, not a gem. connects_to accepts a shards: hash, connected_to takes a shard: argument, and there's a default_shard:
self.default_shard = :default
Every application is already sharded, with one shard. That's worth taking in, because it means adding shards isn't bolting on a new subsystem. It's using an axis that has been there all along, unused.
Key takeaways
- Horizontal sharding splits rows across databases. Every shard has the same schema and a different subset.
- A replica holds everything and solves read load. A shard holds a slice and solves writes and volume.
- Losing a replica costs capacity. Losing a shard takes those users offline.
- Short of multi-primary replication, sharding is how you get more than one machine accepting writes for one table.
- Rails treats shards as first-class:
connects_to shards:,connected_to(shard:)and adefault_shardof:default.
Keep going with lesson 2
The shard key decides everything
Explains why the shard key comes from access patterns and why user_id fits this application better than course_id.
A free account also gives you the quizzes, your progress and notes, and the discussion.