MongoDB change stream replica-set limitation

Question

What is the reasoning behind only making change streams available on replica sets?

kevinadi · Accepted Answer · 2018-01-19T04:16:53.083

24

Change streams implementation is based on the oplog, which is only available on replica sets. Another reason is that a replica set contains a superset of the features of a standalone node, and is the recommended MongoDB deployment for production applications. Hence, it makes sense to implement the change stream feature based on the recommended production deployment topology.

Another major reason is that change streams will output documents that will not be rolled back in a replica set setting (see Rollbacks During Replica Set Failover), so the use of majority read concern is a requirement.

Note that change streams are also available in a sharded cluster, and also a single-node replica set (i.e. a replica set with only one member, although this setup is generally not recommended).

The high-level description of change streams is available in the Change Streams page

The design of change streams is outlined in SERVER-13932.

edited Jan 19 '18 at 04:16

answered Jan 08 '18 at 06:36

kevinadi

13,365
3
33
49

Thanks for that link. – Daniel F Jan 08 '18 at 12:04
3

Just to comment: when I set up a small 20 MB Database without replication, to hold some current data about weather sensors and the like, I need to create a Replica Set which basically forces me to create an unnecessary 1GB oplog, just to be able to subscribe to Change Streams :( – Daniel F Sep 20 '18 at 10:28
1

I just want to point out that **you can set the oplog size via the command line** as well as in the replication options. No need to add secondaries. I thought that the minimum value given in the docs was the one which always would be used. I have successfully created one with `--oplogSize 50` and the oplog has a size of 23 MB. – Daniel F Mar 13 '19 at 09:34
I have rather large databases (size in GBs), does converting them to replica set mean I have redundancy storage that I don't need ? – user1502776 Jun 12 '19 at 21:15
1

"although this setup is generally not recommended" .. thanks. Is there a reason for that? The change stream API is a very useful API even in some applications that do not want/need the resiliency of multiple nodes. – Blaze Oct 19 '20 at 10:14
@Blaze a single node replica set is "generally not recommended" for production use. It's because a replica set provides redundancy, and having a single node provides no redundancy at all. If that node goes down, there are chances that your data can go poof e.g. due to hardware failure. It's perfectly fine for dev purposes, or if the data you're dealing with is not super important (e.g. caching). Depends on your use case, really. – kevinadi Oct 19 '20 at 22:22
I mainly using it for real time streaming. It's not really meant to keep the data. Capped/Tailable collections would have worked, but their blocking semantics aren't there. – Blaze Oct 20 '20 at 17:51

Roman · Answer 2 · 2020-03-14T21:10:15.220

My environment was Windows and the following steps helped me:

Find "C:\Program Files\MongoDB\Server\4.2\bin\mongod.cfg"
Add the following code to the mongod.cfg and options, that you need. More information: Mongo replication Options
```
replication:
 replSetName: "rs0"
```

Restart MongoDB process:

Windows->Task Manager->Services->MongoDB [run restart]

Run in the console mongo 127.0.0.1:27017
Then run rs.initiate()

As a result you are lucky to have something like the following:

> rs.initiate()                                                                               

        "info2" : "no configuration specified. Using a default configuration for the set",    
        "me" : "127.0.0.1:27017",                                                             
        "ok" : 1,                                                                             
        "$clusterTime" : {                                                                    
                "clusterTime" : Timestamp(1584218777, 1),                                     
                "signature" : {                                                               
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),                   
                        "keyId" : NumberLong(0)                                               
                }                                                                             
        },                                                                                    
        "operationTime" : Timestamp(1584218777, 1)                                            
}                                                                                             
rs0:SECONDARY>

macOS version: The configuration file can be found at: `/usr/local/etc/mongod.conf` To restart MongoDB: `brew services restart mongodb-community` — Iddan Aaronsohn, May 05 '20 at 12:27

MongoDB change stream replica-set limitation

2 Answers2