Survival goals

Which failures a cluster survives depending on how many nodes it has and where they are.

A cluster survives the failures that leave every range with a majority of its replicas. This page says what that means depending on the topology.

How it works

Each range is a Raft group. It accepts writes and answers reads while a majority of its replicas answer; if its leader goes down, the others elect another. Group 0, the metadata group, follows the same rule.

What it means for your application

With three nodes on three machines, the cluster keeps serving if one goes down. If two go down, the ranges with their majority on them stop accepting writes until they come back. Your drivers retry the operation in flight when a leader goes down.

Limits

The majority is counted per range. If the replicas of a range are in a single zone or data centre, the loss of that zone leaves it without service even though the rest of the cluster works.

Learn more