How to query large sets of data in Mongoosejs

Question

I am trying to query a large set of users with a specific property(give me all users who are enrolled into a service). The number is expected to be > 10,000. But each user object is huge. In the below function, I am able to log 'reached here' to the cmd line, showing that the function is atleast called, but I cant have access to allUsers, the string 'all users length' isnt even logged to cmd line. My guess is maybe mongoose has a read timeout for large sets of data. Users is a mongoose collection. Please anyone who might have a idea what goings on or has a better way to query large sets of data, all suggestions welcome. Thanks

function get_userIds_ready_for_fulfillment(Users) {
  logger.info('reached here');
  Users.find({ "isEnrolled": true }, (error, allUsers) => {
    if (error) {
      return logger.error('Fullfiment_job_Error', 'error querying all users');
    }
    logger.info('all users length', allUsers.length);
 });
}

well, it depends on the size of your collection, but adding an index on that field will definitely make the query faster and could solve your "timeout" issue if that's what's happening. — maxdec, May 30 '16 at 18:57

score 4 · Answer 1 · edited May 23 '17 at 12:31

You can consider

Using projections to limit the fields returned, thus shrinking the result set. One way to do this is: Users.find({"isEnrolled": true}).select({myField: 1}).exec((err, allUsers) => { ... }). Docs
Using streams/cursors. For example, it is straightforward to stream a result set to a client in node.js. Docs
Paging through the data with skip and limit. Skipping is an expensive operation in the database (discussed here) but is better than what you have now. It is also possible by this approach to omit data if something is inserted past while you're paging.
Using any of the aggregation operators. Docs. (count is also sort of an aggregation operator not listed in that section.)

How to query large sets of data in Mongoosejs

1 Answers1