LSM storage and MVCC

Each core stores its data in an LSM. Each document keeps several versions with their timestamp (MVCC).

How it works

write ──▶ Raft log ──▶ memtable ──(full)──▶ SST file
                                              │
                               compaction ◀───┘  (merges SSTs and drops old versions)

A write goes into the Raft log and into an in-memory table, the memtable. The Raft log is also the LSM's write-ahead log. So each write reaches the disk once. When the memtable fills up it is flushed to an immutable, sorted SST file, and compaction, which runs continuously without anyone asking for it, merges those files and drops the versions that no read and no open transaction can see any more.

Each version carries its HLC timestamp. A read at a timestamp sees the last version before it.

What it means for your application

snapshot reads and transactions read a consistent picture without blocking writes or waiting for them, because each one looks only at the versions that existed at its timestamp. There is no need to run compact.

Limits

Versions are kept 24 hours by default (teselaHistoryRetention). A read at an earlier timestamp fails with SnapshotTooOld.

Learn more

• Hybrid clocks
• Consistency and isolation