Bulk mongodb insert in Meteor or Node

Question

MongoDB supports bulk insert http://docs.mongodb.org/manual/core/bulk-inserts/

I have tried it in Meteor collection:

Orders.insert([
    { "cust_id" : "A123", "amount" : 500, "status" : "A", "_id" : "iZXL7ewBDALpic8Fj" },
    { "cust_id" : "A123", "amount" : 250, "status" : "A", "_id" : "zNrdBAxxeNZQ2yrL9" },
    { "cust_id" : "B212", "amount" : 200, "status" : "A", "_id" : "vev3pjJ8cmiDHHxe4" },
    { "cust_id" : "A123", "amount" : 300, "status" : "D", "_id" : "BBLngRhS76DgeHJQJ" }
]);

but it creates just

{ "0" : { "cust_id" : "A123", "amount" : 500, "status" : "A", "_id" : "iZXL7ewBDALpic8Fj"},
"1" : { "cust_id" : "A123", "amount" : 250, "status" : "A", "_id" : "zNrdBAxxeNZQ2yrL9" }, 
"2" : { "cust_id" : "B212", "amount" : 200, "status" : "A", "_id" : "vev3pjJ8cmiDHHxe4" }, 
"3" : { "cust_id" : "A123", "amount" : 300, "status" : "D", "_id" : "BBLngRhS76DgeHJQJ" }, 
"_id" : "6zWayeGtQCdfS65Tz" }

I need it for performance testing purposes. I need to fill and test database with thousands of testing items. I do inserts in foreach, but it takes too long to fill database.

Is here any workaround? Or can we expect Meteor will support this in next versions?

if it's taking too long you can run multiple clients in parallel each sending documents from a forEach loop - it should speed it up by a factor close to the number of client processes. — Asya Kamsky, Nov 03 '13 at 20:10
See my answer to [why-does-loading-data-into-a-meteor-collection-take-so-long](http://stackoverflow.com/questions/19549337/why-does-loading-data-into-a-meteor-collection-take-so-long/). — David Weldon, Nov 03 '13 at 20:16

Dave · Accepted Answer · 2013-11-17T19:31:13.517

4

You could use exec (nodejs docs) to run a mongo script inside of meteor inside of a Meteor.startup on the server.

Example:

Meteor.startup(function () {
    var exec = Npm.require('child_process').exec;
    exec('mongo localhost:27017/meteor path-to/my-insert-script.js', function ( ) {
       // done
    });        
});

Not optimum, but I think it's your best bet for now. You can also use the command line option --eval against Mongo in exec and pass the insert statement as a string to exec. That might look like this:

Meteor.startup(function () {
    var exec = Npm.require('child_process').exec;
    exec('mongo localhost:27017/meteor --eval \'db.Orders.insert(' + JSON.stringify(arrOfOrders) + ')\'', function ( ) {
       // done
    });        
});

edited Nov 17 '13 at 19:31

answered Nov 07 '13 at 18:10

Dave

2,506
2
20
27

Thanks, I have used the second variant. But double quotes must be single because JSON.stringyfy() adds double quotes and causes `SyntaxError: Unexpected end of input`. – Tomas Hromnik Nov 17 '13 at 17:18
Glad I could help. I hadn't tested either solutions. I'll update my answer to be more syntactically correct. – Dave Nov 17 '13 at 19:30
Consider using mongoimport: http://stackoverflow.com/questions/15365747/how-to-use-mongoimport-with-my-meteor-application-database – Reimund Dec 03 '13 at 23:22

score 3 · Answer 2 · answered Nov 04 '13 at 00:38

When inserting a lot of data into the DB, e.g., in a forEach loop, you want to make sure that there is no reactive content on your page that depends on it. Otherwise the reactive rerendering is going to slow your client down tremendously. You can easily insert several thousand document into a collection in a fraction of a second when all templates are disabled, while the same operation can take several minutes with your CPU at 100% on both the client as the server if there is relevant reactivity happening.

You may want to add a condition to any template whose content depend on this data such as:

Template.myTemplate.items = function() {
  if (Session.get("active")) {
      return Order.find();
  }
}

Then you can deactivate all reactive rerendering before your forEach loop and reactivate it again afterwards (Session.set("active", false)).

This could help, but I don't want to rewrite every template. — Tomas Hromnik, Nov 17 '13 at 17:27

score 0 · Answer 3 · answered Nov 27 '20 at 21:53

You could use rawCollection which is node mongodb driver implemetation in Meteor.Collection.

await Orders.rawCollection().insertMany(arrOfOrders)

It works on 70M data in my case. (await makes it synced so you should consider to use it or not for your purpose.)

Bulk mongodb insert in Meteor or Node

3 Answers3

Linked