Troubleshoot the cluster
Nodes that do not join, lost leaderships and ranges without a majority.
These are the cluster problems seen most often. For each one you have the cause, how to confirm it, how to fix it and what to do so it does not happen again.
A new node does not join the cluster
You start the binary on the new machine with --join and its token, but teseladb admin nodes does not list it and the
node gets no replicas.
Cause
There are two possible causes. The enrolment token was already used or has expired, or port 27100 is closed between the
new machine and the one of the node that --join points to.
How to confirm it
Read the log of the new node. If it says the token was rejected, the problem is the token; if it says it cannot reach
the --join address, it is the network.
Fix
Create a new token with teseladb admin token create, or open port 27100 between the two machines, and start the node
again with --join.
How to avoid it
Create the token right before starting the node, use a different one for each node and check port 27100 when you prepare the machine.
A node gives up all its leaderships
The node stays in the cluster and keeps its replicas. However, teseladb admin ranges shows that it no longer leads any
range and the other nodes take all the write load.
Cause
The clock of that machine has drifted more than 125 ms from the cluster median, and a node outside that margin gives up its leaderships.
How to confirm it
Look in the node's log for the line that notes its clock is outside the margin. It usually shows up several times in a row.
Fix
Check NTP on that machine. As soon as the clock is back within 125 ms of the median, the node takes leaderships again and the balancer gives part of them back to it.
How to avoid it
Keep NTP on every machine of the cluster and add an alert on its drift, so you find out before a node goes past the margin.
A range stops taking writes
Reads of some documents keep working while writes on them fail or wait until they run out of time. All those documents live in the same range.
Cause
The range lost the majority of its replicas, for example because two of the three machines that held them failed at the same time.
How to confirm it
Run teseladb admin ranges --collection <database.collection>. The affected range shows up without a leader.
Fix
Start again the nodes that held its replicas. Once the majority is back, the range elects a leader and takes writes; if those machines cannot come back, restore a backup on a new cluster.
How to avoid it
Deploy on three machines at least and spread their possible failures: power supplies, network switches and, if you can, different data centres.