1

quick question about a mongodb db and collect i've setup and am trying to use python to update each of the documents.

Basically I have a collection of about 2.6million postcode records and have my python script that takes data from a csv file using the postcode as the key.

All the postcodes are unique and both the DB and the CSV have the corresponding keys. The data I'm importing doesn't really matter per se this is more of an exercise to find the best method using python to update my document as I will be planning to do something later with more meaningful data. I've added an index to the postcode column within the mongo collection but this hasn't seemed to speed up the processing.

When I run the code below it seems to take about 1 second per document to update, and as you can guess thats way to long to wait to have all these records updated. Does anyone know of a quicker way to do this and if theres anything from my example below that may be preventing it from running faster.

Any help would be greatly appreciated. Sorry if this is the wrong place I'm not sure if its a mongo issue or a python issue.

Thanks

please find example of the python code i'm using to update the mongo records.

for key, val in testdict.items():
    mycol.update_one({"Postcode": key}, {"$set": {"SOAExample": val}})
    count = count+1
    print(count, " out of ", totalkeys, " done")
OutlawsGalaxy
  • 71
  • 4
  • Possible duplicate of [Bulk update in Pymongo using multiple ObjectId](https://stackoverflow.com/questions/35480660/bulk-update-in-pymongo-using-multiple-objectid) – Mani Mar 25 '19 at 05:29

1 Answers1

0

Look at the bulk_write API which will allow you to batch updates so that you reduce the number of round trips to the server. Also, split your data and run many update processes in parallel so that updates happen in parallel. the database server may be slowish for any particular update due to write_concerns etc. but it can process many updates in parallel.

Joe Drumgoole
  • 1,268
  • 9
  • 9