NodeJs: Filter mongo query results with results from another collection

Question

I have a situation where I need to perform logic on a distinct set of values from a mongo collection (A) and then save result to another collection (B). However the contents of (A) will change over time and so I only want to perform the logic on those documents in (A) where there is not a corresponding document in (B). As joins aren't supported, I am trying to do this at the Node level. I am querying all items in collection (A) and using findOne to look for the corresponding entry in collection (B). If I find it, I would like to remove it from the array, but I am stuck because findOne uses an asynchronous callback which doesn't seem to work with the array filter method. Is there a better way to do this:

function loadNewDocumentsFromDB(callback){
  db.collection('A', function(err, acollection){    
    bcollection.find().toArray(function(err, arr){
      if(arr){
        // toQuery is the global array to be used elsewhere
        toQuery = arr.map(function(config){
          transformed = 
          {
            args: config._id, // these args are a doc representing a unique entry in 'B'
            listings: config.value.id.split(',') // used by other functions
          };

          return transformed;
        })
        .filter(function(transformed){
          db.collection('B', function(err, bcollection){
          bcollection.findOne(transformed.args, function(err, doc){
            // I want these values returned from the filter function not the callback
            if(doc){
              return false; // want to remove this from list of toQuery
            }else{
              return true; // want to keep in my list
          });
        });
      }
      callback();
    });
  }); 
}

score 1 · Answer 1 · answered May 12 '14 at 17:28

This was how I managed to get it working:

function loadOptionsFromDB(callback){
  toQuery = [];
  db.collection('list', function(err, list){    
    db.collection('data', function(err, data){
      list.find().each(function(err, doc){
        if(doc){ 
          transformed = 
            {
              args: doc._id,
              listings: doc.value.id.split(',')
            };
          (function(obj){       
            data.findOne(obj.args, function(err, found){
              if(found){}
              else{
                toQuery.push(obj);
              }
            });
          })(transformed);
        }else{
      //Done finding
          setTimeout(callback, 20000);
        }
      });       
    });
  }); 
}

score 0 · Accepted Answer · edited May 23 '17 at 12:07

0

A better way would be to do this on the database. Check if 2 executions of http://docs.mongodb.org/manual/core/map-reduce/ would be of any use to you. See Merging two collections in MongoDB for more information

edited May 23 '17 at 12:07

Community

1
1

answered May 09 '14 at 18:44

Zlatin Zlatev

3,034
1
24
32

I think this would work best long term as my data sets will keep growing. Short term I managed to get this working by building the array using the each iterator of the cursor. I'll post my code later. – Comart May 10 '14 at 18:09

NodeJs: Filter mongo query results with results from another collection

2 Answers2