5

I understand that when any watched entity of mongo is altered/added , mongo logs these changes in change streams, to which applications can listen to.

When is a single change stream event cleared , what is the maximum capacity of change stream. Are there any negative scenarios wherein log is deleted from change stream before it was notified to the subscribers due to maximum capacity restrictions.

I could not find any such data in their official page: https://docs.mongodb.com/manual/changeStreams/

Manas Saxena
  • 2,171
  • 6
  • 39
  • 58

1 Answers1

8

Change streams are using the oplog as the basis of its data. Thus the "maximum capacity" is based on the size of your oplog (see SERVER-13932).

As long as you're following the change stream, you should be notified of changes in real time. The size of the oplog is not an issue in this case, since you're consuming the change as it happens.

However, a change stream can be resumed using a resume token that points to a specific timestamp in the oplog. When the oplog has rolled over, this resume token is now invalid. Attempting to resume a change stream using an invalid resume token will result in an error:

resume of change stream was not possible, as the resume token was not found. {_data: <the invalid resume token>}

At this point, it's up to the application on how to proceed.

kevinadi
  • 13,365
  • 3
  • 33
  • 49
  • I have an ever increasing data usecase with no deletions only inserts and updates. I listen change stream to burst my application in-memory cache .Since the application is distributed and autoscalable , the number of listeners of change stream is dynamic,hence I do not want to drive deletions of events in change stream from my application. I fear that over a period of time my mongo oplog would fill up. What is the strategy mongo uses in case when op log fills up ? I would prefer to have FIFO deletion strategy for my use case – Manas Saxena Jun 05 '19 at 10:29
  • Have a read through https://docs.mongodb.com/manual/core/replica-set-oplog/ it explains everything you need to know about the oplog. The oplog is basically a circular buffer, so it's basically a FIFO structure. – kevinadi Jun 05 '19 at 21:48