Read older documents - like `tail --follow` - but for a mongodb collection

Question

I am familiar with the MongoDB oplog and change-streams, like so:

  const changeStream = d.db(dbName).collection(dbCollName).watch();

  changeStream.on('change', next => {
    log.info('doc:', next.fullDocument);
  });

but my question is, is there a way to get the most recent 100 documents and then read for changes without missing and documents? Just like tail -f but for a mongodb capped collection?

score 0 · Answer 1 · answered Oct 04 '19 at 16:25

As far as I know, this isn't supported by change-streams, so you'd need to write it manually. Something like the following should first fetch the last 100 documents from the database & log those documents before beginning to log the documents that the change stream sees.

  const conn = await mongodb.connect("mongodb://mongo:27017")
  const db = conn.db("test");
  const coll = db.collection(collName);
  // We want to start watching before doing the fetch of the last 100 docs
  const changeStream = coll.watch();
  let changeStreamDocs = [];
  let hasLoggedInitialDocs = false;
  // we need to make sure we don't log the same doc twice if we see if in the change stream & in the `find`
  const initialDocIds = {};
  function logChangeStreamDocs() {
    // We need to wait for `coll.find` to complete before we can start logging the change stream docs
    if (!hasLoggedInitialDocs) return;
    for (const doc of changeStreamDocs) {
      if (initialDocIds[doc._id]) continue;
      console.log(doc);
    }
    changeStreamDocs = [];
  }
  changeStream.on('change', next => {
    changeStreamDocs.push(next.fullDocument)
    logChangeStreamDocs();
  });
  // find the most recent 100 docs. _id: -1 is descending
  const docs = await coll.find({}, { sort: { _id: -1 }, limit: 100 }).toArray();
  for (const doc of docs) {
    console.log(doc);
    initialDocIds[doc._id] = true;
  }
  loggedInitialDocs = true;
  logChangeStreamDocs();

note: I don't have a replica set handy, so I haven't actually tested this. There's probably a few typos in there

Read older documents - like `tail --follow` - but for a mongodb collection

1 Answers1