8

I have a MongoLab cluster, which allows me to use Oplog tailing in order to improve performances, availability and redundancy in my Meteor.js app.

Problem is : since I've been using it, all my publications take more time to finish. When it only take like 200ms, that's not a problem, but it often takes much more, like here, where I'm subscribing to the publication I described here.

This publication already has a too long response time, and oplog observations are slowing it too, though it's far from being the only publication where observing oplog takes that much time.

Could anyone explain to me what's happening ? Nowhere where i search on the web I find any explanation about why observing oplog slow my publication that much.

Here are some screenshots from Kadira to illustrate what I'm saying :

You can see that oplog observation take a lot of time

Here is a screenshot from another pub/sub :

enter image description here

And finally, one where observing oplogs take a reasonable time (but still slow my pub/sub a bit) :

enter image description here

Community
  • 1
  • 1
David Panart
  • 656
  • 6
  • 20
  • Have you seen this? Is it your situation? http://stackoverflow.com/questions/23429049/cost-of-observing-large-collection-with-oplog-tailing – PaulG Mar 10 '16 at 11:00
  • 1
    Yes I know that, but it is not my question. My question is : while oplog tailing should spare ressources and time on my server, why does it takes so much time in a apparently random way on some of my pusub ? Like I show it above, 1100ms + 2800 ms for observing oploags .... It just unacceptable to slow down my publication that much when it should be a way to improve my app. – David Panart Mar 10 '16 at 17:18
  • how many documents are in your collection? are you using an index with `ensureIndex`? Your time to fetch the documents is really high https://docs.mongodb.com/manual/reference/method/db.collection.ensureIndex/ – Dude Jul 26 '16 at 13:04
  • Hi David Panart, Did you find an answer to this problem? – felipecamposclarke Mar 06 '17 at 22:51
  • 1
    Absolutely nothing :/ It just became a small enough problem so that I don't care anymore, there are so much more possibilities now to optimize Meteor projects, such as redis-oplog by Cult Of Coders – David Panart Mar 07 '17 at 14:36

1 Answers1

1

Oplog tailing is very fast. Oplog tailing isn't the issue here.

You're probably doing a lot of things that you don't realize make publications slow:

  • One-by-one document followed by update loops: You're probably doing a document update inside the body of a Collection.forEach call. This is incredibly slow, and the origin of your poor performance in method bodies. Every time you do a single document update that's listened to by hundreds of concurrent connections, each of those need to be updated; by doing a query following by an update one at a time, neither Mongo nor Meteor can optimize and they must wait for every single user to be updated on every change. It's a doubly-asymptotic increase in your performance. Solution: Think about how to do the update using {multi:true}.
  • Unique queries for every user: If you make a single change to a user document that has say, 100 concurrent unique subscriptions connected to it, the connections will be notified serially. That means the first connection will be notified in 90ms, while the last connection will be notified after 90ms * 100 users later. That's the other reason your observeChanges are slow. Solution: Think about if you really need a unique subscription on each users document. Meteor has optimizations for identical subscriptions shared between multiple concurrent collections.
  • Lots of documents: You're probably encoding each thread comment, post, chat message, etc. as its own document. Each document needs to be sent individually to each client, introducing some related overhead. This is the right schema for a relational database, and the wrong one for a document-based database. Solution: Try to hold every single thing you need to render a page to a user in a single document (de-normalization). With regards to chat, you should have a single "conversation" document that contains all the messages between two+ users.
  • Database isn't co-located with your host: If you're using MongoLab, your database may not be in the same datacenter as your web host (which I assume is Galaxy or Modulus). Intra-datacenter latencies can be very, very high, and this is probably the explanation for your poor collection reads. Indices, as other commenters have noticed, might play a role, but my bet is that you have fewer than a hundred records in any of these collections so it won't really matter.
DoctorPangloss
  • 2,994
  • 1
  • 18
  • 22