18

I've been thinking about how to make a real-time web application using nodejs/socket.io/mongodb. The idea is pretty similar to google docs, where objects edited on a page are emitted and rerendered on all client browsers.

What is the best way to go about doing this? From what I've read I can think of 3 ways:

1) Using mongodb oplogs

Add a 'listener' to mongodb collections, rerender parts of page whenever changes are made to collection (cons: slow?)

2) Using local json

Retrieve mongodb data into json file, use fs to edit, save to mongodb and delete json when done (cons: cumbersome to have extra layer between database and actual app)

3) Using purely socket.io

Rerender without storing, save only after all changes have been made (cons: files probably not rendered correctly in all browsers)

Is there a better way to achieve this? (How does google docs work anyway?) Would really appreciate any help anyone can offer!

Community
  • 1
  • 1
Poh Zi How
  • 1,489
  • 3
  • 16
  • 38

6 Answers6

23

We built a real-time app last year, basically a tool for authors to work on the same page where they could add/remove/edit elements (text, images, videos, etc.)

What we used were:

  • Node.js, with Hapi.js framework (express based)
  • Socket.io
  • No MongoDB but the awesome RethinkDB instead, which is real-time by default and basically uses listeners to tell you whenever something has changed. (mongoDB sucks in our opinion, we used it in the past and it feels like "never again", but that's our opinion)
  • React/Redux in order to update the DOM only for elements that have changed, Angular with its both-ways wouldn't have worked well in our opinion, since several users may modify the same page at the same time and therefore re-rendering all the elements would cause to lose the focus.

And honestly, it's pretty awesome how fast it is.

Vadorequest
  • 16,593
  • 24
  • 118
  • 215
  • This is really similar to what I was trying to make! Never heard of rethink before, but rethink and react sounds like exactly what I need. Is there a particular advantage hapi has over express though other than consistency in large teams? – Poh Zi How Jan 06 '17 at 01:04
  • 1
    I like hapi because of its modularity. It's basically Express in a more modular way. You can use either, it doesn't really matter. There is a really good explanation about the key differences there: http://stackoverflow.com/questions/30469767/how-do-express-and-hapi-compare-to-each-other – Vadorequest Jan 06 '17 at 09:13
  • Just came across this post and I was keen do try RethinkDB . Unfortunately the last git updates where done almost a year ago. This let's wonder if it is still developed – Niklas Mar 09 '18 at 10:49
  • @Niklas https://www.rethinkdb.com/blog/2.3.6-release/ They had quite a hard time at some point because the company wasn't making enough money, but they handled it very nicely and it's still open source and maintained. You can read more following the link. – Vadorequest Jun 08 '18 at 17:57
  • if I would want to add Redis to that mix, what would it be good for? because I see lot talking about it when it comes to Real-Time web apps!! – Siempay Jan 13 '19 at 17:40
  • Redis could definitely be used if you need to fetch external data for instance, like if you connect to a DB. If you use a Redis in front of your DB then it'll likely be faster (redis is faster than a db, that's basically it's point) and avoid to overload the DB connections. – Vadorequest Jan 15 '19 at 00:14
3

This is easy to solve without much of complication and saving documents to databases. You should only save document locations. Node has some very awesome features built for this kind of applications. I recommend you to look into these topics:

  • EventEmitters

  • Streams

Node filesystem has classes that you can use to build this for documents:

You can use socket.io to hook up these events to your client application.

Sulejman Sarajlija
  • 414
  • 3
  • 6
  • 1
    The solution will work but, the fs module of node is not one of the most efficient ways of acheiving the goal.. It will definitely save a lot of time if you're developing a Demo product or a POC.. – Bala Abhinav Jan 10 '17 at 05:47
2

I would go with option 1 & 3 but with slight difference. 1. The first option to tail mongoDB opLog is a good one but the overhead becomes very big on the DB where your app will be making millions of transactions. The meteorJS library is already doing this and you can explore them as they are mature and stable to use than writing our own services.

  1. Option 3 to use socket.io. You can actually use socket.io for publishng changes as well as writing to database if you are using rethinkDB which supports native changefeeds. Native changefeeds meaning, everytime there is a write to the table/collection that you want to watch realtime, it gives a callback with the old and new data where you can use socket.io to publish to all clients.
  2. Another way to do it which is a more robust solution is using rabbitMQ like how Paul had mentioned above.
Bala Abhinav
  • 1,328
  • 1
  • 9
  • 15
1

If I were to do this, I'd probably use a blend. Redis or rabbitmq to manage the socket.io connection list to get the publish and subscribe behavior as quick as possible, with a timer job that periodically flushes the writes of the document to the mongodb for longer term persistence, though arguably you could leave all docs in Redis if you wanted.

Sergio Tulentsev
  • 226,338
  • 43
  • 373
  • 367
Paul
  • 35,689
  • 11
  • 93
  • 122
1

"Building a collaborative document editing application" is actually a chapter in the book "Mastering Node.js". They use:

Also, MongoDB recently published a white paper about data streaming with Apache Kafka to achieve real-time capability: https://webassets.mongodb.com/kafka_and_mongodb.pdf

Jay
  • 1,564
  • 16
  • 24
-2

I think syncing data using socketIO is the best way ... Send data to mongo with an emit. Listen for changes in database using the socket and rerender your page with this changes.

You can also visit nodejs realtime mongodb! to see how to automatically sync data with regina.

tutanck
  • 11
  • 4