Why is my mongodb call so slow?

Question

Alright, so I'm building an application based in Node.js and I am using mongoose to handle my connection to mongodb. I have an endpoint that is such:

getTestStream : function(req, res, conditions, callback) {   
  Activity.find()
    .limit(1000)
    .run(function(err, activities) {
      if (err){
        util.sendError(req, res, "Query Error", err);
      } else if (activities) {     
        res.send(activities);
      } else {
        util.send('nope');
      }
  });
}

For some reason this call takes 700ms+ to complete. The same call without even applying a limit made from mongodb shell returns in about 4ms. It seems like such a simple query, so what's slowing it down so much? I'm guessing I've missed something obvious in configuration somewhere, but I have no idea.

Thanks to anyone who can help on this.

Other info:

mongoose@2.6.0
mongodb@2.0.4
node@0.6.9

I'm not extremely familiar with mongoose, so this is just a shot in the dark- Do you need to specify your limit before your find command? Because maybe it's going out and retrieving the records right away at find(), and then you're trying to re-limit it somehow? This looks somewhat similar to your question: http://stackoverflow.com/questions/5539955/how-to-paginate-with-mongoose-in-node-js — Christopher WJ Rueber, May 02 '12 at 17:25
Modifying the query to include the limit as follows does not impact performance in any obvious way. Activity.find({},{limit:1000}) .run(function(err, activities) { — Rob Riddle, May 02 '12 at 17:35
I'd recommend enabling the query profiler: http://www.mongodb.org/display/DOCS/Database+Profiler Is the query properly indexed? — Kyle Banker, May 02 '12 at 20:31
Its not an issue of mongodb being slow, its something with mongoose or node more likely. If I run db.activities.find().limit(1000).explain() inside of the mongo console/shell the query takes less than 5 milliseconds. Also you'll notice I'm not searching for anything, I just want 1000 records from the collection right now, I don't care what I get, so indexes should be irrelevant. I did that intentionally for testing to bypass any possible index issues. — Rob Riddle, May 02 '12 at 21:04
It's not clear to me how you're timing things, but are you sure the bulk of the time isn't in the res.send(activities) call rather than the query itself? — JohnnyHK, May 02 '12 at 21:07
Wes, here's the output of db.activities.stats() http://pastebin.com/GWuSV4vV — Rob Riddle, May 02 '12 at 21:19
@JohnnyHK to check I put console.log(new Date().getTime()) in certain sections of the call. Here's the output of that http://pastebin.com/2wbWt8cZ It appears your theory has some merit as the res.send takes about 300ms, but the db call is still taking around 440ms+ to return what I would expect to be 100ms max. Also on a related note, the return time seems to correlate with how many records I try to retrieve. Its fast for just a few records, but as I increase the limit it gets slower and slower. — Rob Riddle, May 02 '12 at 21:40

score 9 · Accepted Answer · edited May 23 '17 at 12:14

9

After experimenting for a while, I've found several contributions to slowness, hopefully this helps anyone with a similar issue:

The objects I'm requesting are large, so processing them takes some time. For large objects modify the query to only return the fields you need right now.
Mongoose is useful, but it can really slow down when you request a lot of items, its better to just directly interface with node-mongodb-native if you want speed for a call. (This was about a 50%+ speed increase for my scenario)

Using these techniques I can now process 4000 records in less time than I was processing 1000 before. Thanks for anyone who commented, and special thanks to Gates VP for pointing out that mongoose wasn't really a good fit for this kind of call.

edited May 23 '17 at 12:14

Community

1
1

answered May 04 '12 at 00:14

Rob Riddle

3,544
2
18
26

1

How large were the objects you were querying? Just would be nice to have a rough idea of how large an object might need to be to cause the perf problem you were seeing. – Giscard Biamby Mar 03 '13 at 17:36
2

They weren't huge, its been a while since I addressed this issue, probably 4 string fields each about 50-100 characters. Keep in mind that now mongoose has a [lean()](http://mongoosejs.com/docs/api.html#query_Query-lean) option which removes most any getter/setter wrapper bloat, as well as [stream()](http://mongoosejs.com/docs/2.7.x/docs/querystream.html) function so that you can get results as they are found. – Rob Riddle Mar 04 '13 at 16:56
Thanks for the tip off for lean. I'm currently retrieving ~280 records but they are nested multiple levels deep, with some sub-documents having 200 elements. It took my mongoose query from 2.5 seconds to 15ms (which is what I was getting in the mongo shell). – Leonidas May 21 '13 at 03:17

score 0 · Answer 2 · answered May 03 '12 at 00:27

0

The same call without even applying a limit made from mongodb shell returns in about 4ms.

The shell applies a limit of 30 or so by default. Try doing from the shell with an actual limit?

Also, you may want to try a .explain() from the Shell.

If none of that works, then you can take @Kyle Banker's suggestion and check out the profiler.

answered May 03 '12 at 00:27

Gates VP

44,957
11
105
108

Not to seem ungrateful, but the problem doesn't seem to lie with mongodb. As I stated in another comment above: "If I run db.activities.find().limit(1000).explain() inside of the mongo console/shell the query takes less than 5 milliseconds." The profiler shows the same results on real calls from the endpoint. I'm assuming its a driver config or something to do with the mongoose wrapper because so far I can find no flaws with the mongodb part. – Rob Riddle May 03 '12 at 01:27
1

So I edited the tags and added Mongoose. If you don't get a response here, I would definitely check out their Google Group. Note that Mongoose *is* building an object to wrap around every result that you are pulling out of the DB. So it's quite possible that Mongoose is simply not optimized for that. – Gates VP May 03 '12 at 17:03
Ah good call. I thought I had added mongoose but apparently not. I tested this same function with different (smaller) database objects in another collection and it appears about 100ms faster, so it could directly relate to how complex the db record is as to how much time mongoose takes to wrap each record on retrieval. – Rob Riddle May 03 '12 at 20:53

score -1 · Answer 3 · answered Apr 20 '13 at 06:32

-1

checkout ensureIndex This will speed up your search

answered Apr 20 '13 at 06:32

nickleefly

3,733
1
29
31

Why is my mongodb call so slow?

3 Answers3

Linked