0

when i am trying to fetch documents from my collections with aggregate joins, i got an error "aggregation result exceeds maximum document size (16MB)" and i am also using "{allowDiskUse: true}" in my code but again it shows an error. please let me know anyone how to solved it?

Siva Mandapati
  • 61
  • 2
  • 10
  • You cannot increase the size, it's a hard limit. You didn't listen to what you were told on [the previous question](http://stackoverflow.com/q/36231528/5031275). You cannot try to `$push` every single document in your collection into an array. Which do doubt you were trying to do when you go this error. That is not how you go about counting total results for paging. – Blakes Seven Mar 26 '16 at 09:10
  • what i am trying to do is , joining 2 collections as below, db.collection("bags", function (err, bags) { bags.aggregate([{$sort: {todayDate: -1}}, {$match: {}}, {'$lookup': { from: "donor", localField: "regNo", foreignField: "regNo", as: "donordata" } }, {$unwind: {path: "$donordata", preserveNullAndEmptyArrays: true}} ], {allowDiskUse: true}, function (err, info) { res.json(info); }); }); where i am mistake? – Siva Mandapati Mar 26 '16 at 09:44
  • 1
    Possible duplicate of [How could I write aggregation without exceeds maximum document size?](http://stackoverflow.com/questions/29644587/how-could-i-write-aggregation-without-exceeds-maximum-document-size) – Frederick Cheung Mar 26 '16 at 09:55
  • Thank you @Frederick Cheung, but when i am trying $out: "collection" i got other error "duplicate key error index mongodb". – Siva Mandapati Mar 26 '16 at 10:04
  • @SivaMandapati Just use the cursor output for now, which should be the actual default if you are working in the shell. The `$unwind` will duplicate the `_id` so another alternate is a last stage `$project` where you remove the `_id` field. `"_id": 0` – Blakes Seven Mar 26 '16 at 10:06
  • @BlakesSeven Please let me know where should i remove _id in the last mentioned code – Siva Mandapati Mar 26 '16 at 10:29

1 Answers1

1

As stated, you are blowing up the "response" size limit because you are not returning a "cursor" and .aggregate() is instead trying to return the result in a single BSON document.

So instead, use the "cursor" option which produces a stream interface you can then act on the events from: 

 db.collection("bags", function (err, bags) { 

  var result = [];

  var cursor = bags.aggregate(
    [
      { "$sort": { "todayDate": -1 }}, 
      //{ "$match": {}}, 
      { "$lookup": { 
        "from": "donor", 
        "localField": "regNo", 
        "foreignField": "regNo", 
        "as": "donordata" 
      }}, 
      { "$unwind": { "path": "$donordata", "preserveNullAndEmptyArrays": true}} 
    ], 
    { 
      "allowDiskUse": true,
      "cursor": { "batchSize": 20 }
    }
  );


  cursor.on("data",function(data) {
    result.push(data);
  });

  cursor.on("end",function() {
    res.json(result);
  })

});

Ideally you should be using the stream on writing ouput lines as well, such as with a stream writer for JSON. But for example purposes we are just appending data into an array for each cursor result that is iterated.

Blakes Seven
  • 49,422
  • 14
  • 129
  • 135
  • Thank's a lot @Blakes Seven, it was working.. it was taking 20 seconds to fetch 48K records. how can i optimize this – Siva Mandapati Mar 26 '16 at 11:55
  • @SivaMandapati If you have more questions then please [Ask Another Question](http://stackoverflow.com/questions/ask) that is how it works here. Please do a bit of research though. – Blakes Seven Mar 26 '16 at 12:14