I have several large collections in a mongo db. some with sizes ~500 MB.
(Collections get bigger over time reaching the 500MB in size and bigger)
Collection1:
109091 documents
total size: 154.3MB
avg size of document: 1.4KB
Collection2:
102197 documents
TOTAL SIZE
15.1MB
AVG. SIZE
155B
Collection3:
319 documents
TOTAL SIZE
115.8KB
AVG. SIZE
372B
Collection1 relates to collection2 on the acc field Collection1 relates to Collection3 on the tc field
Only index in each of the collection its the record _id which I don't really use to query the collections.
I am running node js on my local machine: windows 10 RAM: 16 GB
The mongoDB database is in a linux machine which I dont have access to.
I am trying to iterate through one of the collections and process each record while fetching the corresponding record on another collection.
processStream : function(){
var stream = Collection.find().cursor();
stream.on('data', function (doc) {
if (doc.tn !== null && (doc.tc !== null || doc.cTc !== null) && doc.acc !== null){
//function that findOne per the account field
module.exports.getAcc(doc.acc, function(res){
if ( res!== null ){
//function that findOne per the tc field
module.exports.getProd(doc.tc, function(res2){
if (res2 !== null){
//write to another mongodb collection
}
});
}
});
}
}).on('error', function (err) {
console.log(err);
}).on('close', function () {
console.log('processing finished');
callback();
});
}
Then in the routes file I have the following call
app.get('/api/process/', function(req, res){
console.time('proc');
functions.processStream(function(){
console.timeEnd('proc');
console.log('...operations ended');
});
res.json('...processind started');
});
This takes for a long time. The bigger the collection the longer it takes. Is there any other way to loop over a collection in mongoDB and process each of its records any faster? I am sure node/mongodb/mongoose can be used for a lot larger collections (GB?? maybe)..
Taking into consideration the size of my collection just looping through the records in the collections it took 19 minutes.