Mongoose insertMany limit

Question

Mongoose 4.4 now has an insertMany function which lets you validate an array of documents and insert them if valid all with one operation, rather than one for each document:

var arr = [{ name: 'Star Wars' }, { name: 'The Empire Strikes Back' }]; Movies.insertMany(arr, function(error, docs) {});

If I have a very large array, should I batch these? Or is there no limit on the size or array?

For example, I want to create a new document for every Movie, and I have 10,000 movies.

score 23 · Accepted Answer · answered Mar 16 '16 at 18:09

I'd recommend based on personal experience, batch of 100-200 gives you good blend of performance without putting strain on your system.

insertMany group of operations can have at most 1000 operations. If a group exceeds this limit, MongoDB will divide the group into smaller groups of 1000 or less. For example, if the queue consists of 2000 operations, MongoDB creates 2 groups, each with 1000 operations.

The sizes and grouping mechanics are internal performance details and are subject to change in future versions.

Executing an ordered list of operations on a sharded collection will generally be slower than executing an unordered list since with an ordered list, each operation must wait for the previous operation to finish.

this is not up to date - starting with v3.6 the batch size is 100,000 — Elad Katz, Oct 07 '20 at 08:27

score 18 · Answer 2 · answered Feb 14 '18 at 15:05

18

Mongo 3.6 update:

The limit for insertMany() has increased in Mongo 3.6 from 1000 to 100,000. Meaning that now, groups above 100,000 operations will be divided into smaller groups accordingly. For example: a queue that has 200,000 operations will be split and Mongo will create 2 groups of 100,000 each.

answered Feb 14 '18 at 15:05

FireBrand

463
5
16

I tried adding 50000 at once but sadly it was slow. I tried with 1000 and it was faster. – Ashok Dey May 30 '18 at 17:51
1

@AshokDey, i never said it would be faster :) never the less - that might depend on the type of your machine(amount of RAM, disk writes etc...). Are you sending the data to a different server using `async/await`? that also might cause it to be slower. – FireBrand Jun 03 '18 at 16:22
Ahh.. seems like I forgot to paste the URL of the Github repo, here it is: https://github.com/ashokdey/scale You can know about the use case in the Readme and I would be happy to see you as a collaborator. For my use case, it was quicker compared to 50k insert at a time. – Ashok Dey Jun 04 '18 at 17:31
1

Is there some limit on the total size of the documents too? – KaliCharan May 09 '19 at 07:15
@KaliCharan Yes, according to MongoDB's official docs the size limit for every document is 16MB, see here: https://docs.mongodb.com/manual/reference/limits/ You can increase the size limit (see the same link) but for storing such large documents MongoDB might not be the optimal choice and you would rather use a cloud storage of some sort (S3, GCS etc...). – FireBrand Oct 08 '20 at 08:24

score 7 · Answer 3 · edited Mar 14 '21 at 01:02

The method takes that array and starts inserting them through the insertMany method in MongoDB, so the size of the array itself actually depends on how much your machine can handle.

But please note that there is another point, which is not a limitation but something worth keeping into consideration, on how MongoDB deals with multiple operations, by default it handles a batch of 1000 operations at a time and splits whats more than that.

Mongoose insertMany limit

3 Answers3

Linked