Mongo update using forEach is too slow

Question

I have 2000 documents, and I want to add a new field (newUUID) to each one and set its value based on an existing one (oldUUID).

I'm running the following query:

var cursor1 = db.myCollection.find();
cursor1.forEach(function(results){ db.myCollection.update({_id: results._id}, {$set: {newUUID: results.oldUUID}})})

But the iteration takes a few long minutes

MongoDB server version: 3.6.0

MongoDB shell version v4.4.1

Starting in MongoDB 4.2 you can use `updateMany()` with aggreagtion https://docs.mongodb.com/manual/tutorial/update-documents-with-aggregation-pipeline/ — deadshot, Apr 21 '21 at 08:09
Use [**`bulkWrite`**](https://stackoverflow.com/questions/54714148/mongoose-update-or-insert-many-documents/54714424#54714424) option instead — Ashh, Apr 21 '21 at 08:20
@Ashh sounds good, but how to do it in Mongo shell without Mongoose? — Alex L, Apr 21 '21 at 12:36
@AlexL You either need to update with the query which is only available in 4.2+ or need to write some sort of code in the language you are working.. — Ashh, Apr 22 '21 at 14:47

score 3 · Answer 1 · answered Apr 21 '21 at 08:10

If you're using Mongo version 4.2+ you can use pipelined updates

db.collection.updateMany(
{},
[
  {
    "$set": {
      "newUUID": "$oldUUID"
    }
  }
])

Mongo Playground

For a lesser Mongo version there is no way to update a document with it's own values ( unless you want to use something like $out ), so you have to read it into memory and update each in code.

Alex L · Accepted Answer · 2021-04-25T11:25:40.467

I managed to solve the issue in the following way: I've made a python script that uses bulk_write although the real issue was the find method I was using - I was using it without filter.

The script:

import pymongo
from pymongo import UpdateOne
import time

conn = pymongo. MongoClient("mongodb://localhost",
                             username='',
                             password='',
                             tls=False, tlsCAFile=None)
db = conn.some_db
collection = db['myCollection']


start_time = time.time()
operations = []
updated = 0

for doc in collection.find({}, {'_id': 1, 'oldUuid': 1}):
    operations.append(
        UpdateOne({"_id": doc["_id"]}, {"$set": {"newUUID": doc["oldUuid"]}})
    )

    # Send once every 1000 in batch
    if len(operations) == 1000:
        collection.bulk_write(operations, ordered=False)
        updated += len(operations)
        operations = []
        print("updated 1000")

if len(operations) > 0:
    collection.bulk_write(operations, ordered=False)
    updated += len(operations)

processed_time = time.time() - start_time

print(f"Updated {updated} documents")

print(f"The operation took {processed_time} seconds")
print(f"The operation took {processed_time/60} minutes")

Mongo update using forEach is too slow

2 Answers2