Change streams

MongoDB change streams work on a collection, a database or the whole cluster. Events come out in commit order, with transactions together, with no losses or duplicates even when ranges split, merge or change leader.

const stream = db.orders.watch([{ $match: { operationType: { $in: ["insert", "update"] } } }]);
for await (const change of stream) {
  console.log(change.operationType, change.documentKey);
}

Resume

Every event carries a resume token. If your application goes down, open the stream again with resumeAfter or startAfter and carry on where it stopped. You can resume while the change is still in the history the cluster keeps: 24 hours by default, which you change with the teselaHistoryRetention parameter.

db.adminCommand({ setClusterParameter: { teselaHistoryRetention: { seconds: 172800 } } });

Pre-images and post-images

Turn on changeStreamPreAndPostImages on the collection and ask for fullDocumentBeforeChange or fullDocument: "updateLookup" as in MongoDB. changeStreamOptions.preAndPostImages.expireAfterSeconds limits how long they are kept.

DDL and large events

Create, drop and change events for collections and indexes come out with showExpandedEvents, and an event larger than 16 MiB is split with $changeStreamSplitLargeEvent.