One Raft group per range

In TeselaDB, each key range is a Raft group of its own. Its replicas agree on the order of their writes among themselves, without going through a central coordinator. Here is how it works and what it means for your application.

What a range is

A range is an interval [start, end) of the ordered key space. A collection starts in one range and splits when it goes over 64 MiB of live data or 512 MiB on disk with its versions, or when it takes too much load. Each range keeps its replicas on several nodes and is the unit that the cluster replicates and places, which is also the unit that splits when it crosses one of those limits.

How a write is committed

One replica of each range is the leader. It proposes the writes in its log, and an entry is applied once a majority of the replicas have it on disk. With its lease, the leader answers reads without asking the others. A follower can serve reads up to its closed timestamp, which the leader publishes 3 seconds behind its clock.

   range 42                 range 43
 ┌─────────────┐         ┌─────────────┐
 │ leader  n1  │         │ leader  n2  │
 │ replica n2  │         │ replica n3  │
 │ replica n3  │         │ replica n1  │
 └─────────────┘         └─────────────┘

Many groups on each core

A node carries many ranges. Each core of each node runs many groups at once, and its Raft log is also the write-ahead log of the storage, so a write is not stored twice on disk. The cluster metadata, such as the nodes, the catalogue and the list of ranges, lives in another Raft group, Group 0.

What you get

Ranges are independent. A problem in one does not slow the others down. If a node fails, each range it led elects a new leader among its replicas and keeps taking writes while a majority answers. A write confirmed with w: "majority" survives the loss of any minority of its range's replicas.

What it costs

A range needs a majority of its replicas to take writes, so each write waits for the disk of that majority. If you spread the replicas across distant data centres, that wait shows in the latency. More detail in Raft in every range and Ranges and replicas.