node-mongodb Protection against websocket data loss

Question

I am on the client side of a websocket API which uses SocketCluster for its pub/sub.

After authenticating I receive every second json data via

SCsocket.on('authenticate', function(){
   var channel = SCsocket.subscribe('channel1');
   channel.watch(function(data){
      console.log(data); 
   });
});

of the form

[
  {
      "product": "Product1",
      "price":   "10.0" 
  },
  {
      "product": "Product2",
      "price":   "15.0"  
  }
]

Instead of printing the data, I will save it to a mongo db.

In case some of the data cannot be uploaded, I need some sort of a safety net. Something that allows me to upload the data from the websocket to mongo db in retrospect.

What are best practices for doing that? I hope this question is not too broad, I am new to node and mongo db.

Thanks!

score 0 · Answer 1 · answered Jul 27 '17 at 09:35

This is an example of how you can handle the data and save it into Mongo. I didn't have a mongo server running, so you'll have to check that the saving actually works. But the principle is there.

Not sure what you mean by safety net but I added in a check to see if the data is defined, you should more checks in there for your own specific cases.

Let me know if you need any help with something specific.

SCsocket.on('authenticate', function () {

    // subscribe to channel 1
    var channel = SCsocket.subscribe('channel1');
    channel.watch(function (data) {
        // if there is any data
        // you should do more specific checks here
        if (Object.keys(data).length > 0) {

            // connect to mongo
            MongoClient.connect(url, function(err, db) {

                assert.equal(null, err);

                // insert data
                insertDocument(data, db, function() {
                    db.close(); // close db once insert
                });
            });
        } else {
            // handle invalid data
            console.log('data sent is invalid');
            console.log(data);
        }
    });
});


// insert into mongo
var insertDocument = function (data, db, callback) {

    // save into product
    db.collection('product').insertOne(
        // insert data sent from socket cluster
        data, function (err, result) {
        assert.equal(err, null);
        console.log("Inserted successfully into");
        console.log(result); // output result from mongo
        callback();
    });
};

the checks are valid but it's not quite what I meant. I am trying to protect from a situation where the mongo server is down for an hour and all the data the websocket emitted in that time is lost. I guess I would need a second server that kicks in. — tenticon, Jul 27 '17 at 11:38
I would recommend returning a nice `DB down` error to the user through SocketServer if the mongo server isn't running. Why would you want to run the socket server if your database isn't running? This way you don't have to handle a queue of data which could go out of sync with more incoming data. — Luke Brown, Jul 27 '17 at 11:57
unfortunately i cannot decide when the socket server runs. I am on the client side of the websocket — tenticon, Jul 27 '17 at 13:07
Oh! So do you mean: socket server sends data to you and you want to save that data, or queue it for later if your mongodb server is down? — Luke Brown, Jul 27 '17 at 13:13

node-mongodb Protection against websocket data loss

1 Answers1