Copy a huge bunch of documents one collection to another in mongosh

Question

I am trying to copy a million document in a collection to another colletion.

> db.source_collection.find().forEach(function(doc) { db.dest_collection.insertOne(doc)});

This works fine. but I wonder it might cause any trouble to operating Database if I loop a million documents.

Can I give a sleep while looping in mongh shell? Is there any better solution for this?

Please advise me.

See https://stackoverflow.com/questions/11554762/how-to-copy-a-collection-from-one-database-to-another-in-mongodb/67077465#67077465 — Wernfried Domscheit, Mar 06 '23 at 09:11

score 2 · Accepted Answer · answered Mar 06 '23 at 09:06

You can use bulkWrite() rather than insertOne , it will reduce your time significantly.

var bulk = db.dest_collection.initializeUnorderedBulkOp();
var counter = 0;
db.source_collection.find().forEach(function(doc) {
  bulk.insert(doc);
  counter++;
  if (counter % 1000 == 0) {
    bulk.execute();
    bulk = db.dest_collection.initializeUnorderedBulkOp();
  }
});
if (counter % 1000 != 0) {
  bulk.execute();
}

In this code For every 1000 documents, the code executes the bulk operation using the execute() method, and then re-initializes the bulk operation. Finally, after all documents are processed, the code executes the remaining bulk operation.

score 0 · Answer 2 · answered Mar 06 '23 at 11:32

0

If you use the aggregation $out stage, you can copy the documents directly on the server and avoid the network round trip to the client.

db.source_collection.aggregate([{$out:"destination_collection"}])

answered Mar 06 '23 at 11:32

Joe

25,000
3
22
44

Copy a huge bunch of documents one collection to another in mongosh

2 Answers2