5

We're building a real-time API that needs to save data updates specific to a user. We are thinking about opening a change streams on an aggregate representing the data the user should get and serving it over a WebSocket.

I could not find docs on change streams limitations, meaning how many open change streams at the same time will MongoDB support and is it scalable by adding more replicas to mongo.

We are talking about a few thousands of open change streams.

Videsh Chauhan
  • 371
  • 2
  • 18
talarari
  • 539
  • 3
  • 7
  • 19

1 Answers1

0

Each open change stream will have its cost, the top limit for the number of change streams will vary based on the rate of data changes in the collections you are watching. If you are looking to have 1000+ streams it would be better to use some kind of message queue on top of the change streams to serve messages to many consumers (for example there is a connector for Kafka that is based on the change streams). Yes, you could scale by targeting change streams to different members of the replica set, e.g. secondary1 will serve events from collection A and B, secondary2 will serve events from collection C and D. The general recommendation is to keep the number of change streams as low as possible. Each change stream is a connection, you probably don't want to have a connection per user.

Some related posts Severe performance drop with MongoDB Change Streams

Katya Kamenieva
  • 316
  • 1
  • 5