Bulk insert in MongoDB with mongoose for multiple collections

Question

I have 2 collections(data, metaData)

data schema is

{
_id: ......,
name: ......, //not unique
mobile: ......, // unique or null
email: ......, // unique or null
uniqueId: ......, // unique or null
}

at least one of unique data is required for insert

metaData schema is

{
_id: ......,
dataId: ......,//refrence from _id of data collection
key: ......,
value: ......
}

JSON array is getting from client

[{
  name: "abc",
  mobile: 9999999999,
  mData: {
    c1: 123,
    c2: "xyz"
  }
},
{
  name: "qwerty",
  email: 'qwerty@mail.com',
  mData: {
    c1: 123,
    c2: "zxc"
  }
}
......
]

I am iterating through the array and inserting each of them in both collections into MongoDB.

let Bulk = Data.collection.initializeUnorderedBulkOp();
dataArr.forEach(function(item) {
  let data = service.generateData(item);
  // data.query: {mobile: ..., email: ..., uniqueId: ...}
  // if value exists then keys is also exists for mobile, email, uniqueId in query
  Bulk.find(data.query).upsert().updateOne(data.doc);
});
Bulk.execute((e, d) => {
  let metaBulk = MetaData.collection.initializeOrderedBulkOp();
  let length = dataArr.length;
  dataArr.forEach(function(data) {
    Data.findOne(data.query).exec(function(err, data) {
      length--;      
      for(let key in data["mData"]) {
        let value = data["mData"][key] || "";
        let mData = service.generateMdata(key, value, data._id);
        metaBulk.find(mData.query).upsert().updateOne(mData.doc);
      }
      if(length == 0) {
        metaBulk.execute();
      }
    });
  });
});

my solution is working fine right now but it's taking so much time to iterating data collection for finding ids for metaData collection.

I need a way of inserting the data in bulk into MongoDB without find query for data id. Is there any option to perform bulk upserts with mongoose for multiple collections in a single query.

Is the data for each array item not unique already? If so then why not simply apply the already unique identifier from each array? It looks like you are abstracting and in reality your "client" is actually supplying you with uniquely identified items already. So your real problem is letting the driver assign a value for `_id` when you should be using the client supplied value instead. — Neil Lunn, Oct 18 '17 at 06:32
Would it be an option to move all the information into one collection only? So the metadata elements would simply turn into an array inside a data document? That'd be way more of a document-store kind of design. You should avoid any joining concepts if possible. — dnickless, Oct 18 '17 at 19:45
@NeilLunn all data in array is unique but have some keys, I have updated my question according to real data. So its contains multiple column with null or unique data. — Gaurav Kumar Singh, Oct 20 '17 at 05:37
@dnickless no option to move all the info in one collection. — Gaurav Kumar Singh, Oct 20 '17 at 05:41
@NeilLunn i have updated the `data` collection and `JSON` array for more clarification. — Gaurav Kumar Singh, Oct 20 '17 at 05:45
You have not really addressed anything that is commented on. If you read carefully then you should see the concept of an "upsert" means that something in the data needs to be considered "unique" by the query that is looking for it in `.find()`. So your query conditions really identify a "unique key" already. Your whole problem revolves around finding the `_id` values that are "upserted". You resolve the problem by simply using the "already unique" data as that value instead of waiting for the driver to assign it. It's a pretty simple concept. — Neil Lunn, Oct 20 '17 at 06:04
As has also been commented "why is this in two collections?". The provided data is already "related" in single stream, and this is really what MongoDB is for. You are trying to model in a "relational" fashion. And for a database that is not meant to do that, this is really not a good idea. In short, you are creating problems that should not exist by ignoring the fact that natural "embedded" relations in the data already supplied. — Neil Lunn, Oct 20 '17 at 06:06
@NeilLunn we required two collections for the same because `metadata` is not independent collection, for example a metadata key have more than one value in different condition. So its not possible to take both documents in a single collection. — Gaurav Kumar Singh, Oct 24 '17 at 07:35
Dude what can I say other than well done in throwing away 50 rep for nothing. You were asked repeatedly to "show your queries", by myself and others and you simply have not done that. If you cannot grasp what I've already said and realize how wrong what you are attempting is, then I cannot help you. You have several simple fixes here that you just keep ignoring. — Neil Lunn, Oct 24 '17 at 07:39
@Gaurav https://stackoverflow.com/questions/25285232/bulk-upsert-in-mongodb-using-mongoose — Dipak, Oct 24 '17 at 08:33
@Gaurav https://stackoverflow.com/questions/28218460/nodejs-mongoose-bulk-update — Dipak, Oct 24 '17 at 08:37

score 2 · Answer 1 · answered Oct 22 '17 at 04:02

No multiple collection update in a single command for your scenario. In your case if you can include metadata array inside parent collection it can insert data with single command with updateMany(). MongoDB also supports bulk insert through the db.collection.insertMany().

db.data.insertMany( [{ name: "abc",mobile: 9999999999, mData: { c1: 123, c2: "xyz"} },
                                            {name: "qwerty",email: 'qwerty@mail.com',mData: { c1: 123, c2: "zxc" }}]);

Also you can use db.collection.bulkWrite() as well.

score -2 · Answer 2 · answered Oct 19 '17 at 04:46

I think what you can do is:

async.each(jsonArray, function(jsonData,callback){
  //first insert data in data schema
  var data = new data(jsonData);
  data.save(function(err){
    if err throw err;
    //then you save the data in metaData collection
    async.each(jsonData.mData, function(metadata, callback2){
      var metaDataObj = new metaData(metadata);
      metaDataObj.dataId = data._id;
      metaDataObj.save(function(err){
       callback2();
      });
    }, function(err, results1){
      callback();
    });
  });
}, function(err, results){
   console.log('Data is saved');
});

Not a relative answer according to bulk operation for big data — Gaurav Kumar Singh, Oct 20 '17 at 05:42

Bulk insert in MongoDB with mongoose for multiple collections

2 Answers2