Kill a node and watch it recover

A node can fail without warning, after a power cut or with a failing disk. Here you cause it with the equivalent of a kill -9 in the Docker Compose cluster and watch what TeselaDB does with its ranges.

What you will build

Nothing new to leave running: with the three-node Docker Compose cluster, you kill a node, check that you can still write, look at the new leaders and see the node catch up when you start it.

Requirements

• The cluster from A three-node cluster with Docker Compose running.
• mongosh with a connection to that cluster.

1. See where the leaders are

docker compose exec node1 teseladb admin --dbpath /data ranges --collection test.orders

Each range has a replica on each node. One of them is its leader.

2. Kill a node

docker compose kill node2

docker compose kill sends SIGKILL to the container of node2, so the node has no time to close anything.

3. Keep writing

db.orders.insertOne({ customer: "Ana", total: 30 });

The ranges node2 led elect a new leader among the replicas on node1 and node3, which hold the majority. If a write hits that change, the driver retries it.

Try it

Look at the ranges again with the command of step 1. No range has node2 as its leader now, because the ranges it led moved their leadership to node1 or node3. Start the node again and look at them once more:

docker compose start node2
docker compose exec node1 teseladb admin --dbpath /data ranges --collection test.orders

node2 receives what was written while it was down and has up-to-date replicas again. If a node does not come back, the balancer creates the missing replicas on the other nodes.

Clean up

If you will not use the cluster any more, stop the containers and delete their volumes:

docker compose --profile join down -v

Next steps

• Fault tolerance, with what happens in each kind of failure.
• Raft in every range, with how a leader is elected.