How to do a Bulk Insert with MongoEngine?

Question

I am struggling to do a Bulk Insert with MongoEngine. I can easily do a .save() with a loop to write to the database, however I cannot find the procedure to do a Bulk Insert from the MongoEngine documentation. The only mentioning was here on SO: multi document insert using mongoengine into mongodb. I tried to do it this way but I just insert all the documents without validating them in the Schema.

This is my code:

import mongoengine as me

# connect to MongoDB
me.connect(host='CONNECTION_STRING')

# venue Schema
class newVenues(me.Document):
  name = me.StringField(required=True)
  latitude = me.DecimalField(required=True)
  longitude = me.DecimalField(required=True)


# some test data
venues = [{'name': 'Vega',
           'latitude': 55.672867,
           'longitude': 12.473692},
          {'name': 'Tivoli',
           'latitude': 55.681256,
           'longitude': 12.553412}]

# the list of venues to bulk insert to MongoDB
venues_to_insert_list = []
for venue in venues:
    venues_to_insert_list.append(
    newVenues(
    name=venue.get('name'),
    latitude=venue.get('latitude'),
    longitude=venue.get('longitude')
  )
)

# bulk insert to MongoDB
newVenues.objects.insert(venues_to_insert_list)

Yes, but I am not sure how to go about using .insertMany() or with PyMongo I guess it is .insert_many(). I tried doing it but got this error: TypeError: document must be an instance of dict, bson.son.SON, bson.raw_bson.RawBSONDocument, or a type that inherits from collections.MutableMapping. — Benjamin Andersen, Dec 29 '17 at 21:56

score 1 · Answer 1 · answered Dec 29 '20 at 19:12

import mongoengine as me

# connect to MongoDB
me.connect(host='CONNECTION_STRING')

# venue Schema
class newVenues(me.Document):
    name = me.StringField(required=True)
    latitude = me.DecimalField(required=True)
    longitude = me.DecimalField(required=True)


# venue Schema
class newVenues(Document):
    name = StringField(required=True)
    latitude = DecimalField(required=True)
    longitude = DecimalField(required=True)


# some test data
venues = [
    {"name": "Vega", "latitude": 55.672867, "longitude": 12.473692},
    {"name": "Tivoli", "latitude": 55.681256, "longitude": 12.553412},
]

# the list of venues to bulk insert to MongoDB


venues_to_insert_list = [newVenues(**data) for data in venues]

# bulk insert to MongoDB
newVenues.objects.insert(venues_to_insert_list)

This answer just reformats the OP's code but replaces the loop with list comprehension. It does not answer the question of how to do a bulk insert _with_ schema validation. — craigim, Sep 21 '21 at 18:51

score 0 · Answer 2 · answered May 21 '20 at 19:53

If you want to insert all the documents without validating them in the Schema. You need to create dynamic document in mongoengine

Below code is for read csv file and insert in mongodb in collection without validate schema [schema is also make in dynamic way].

import mongoengine as me
class DynamicDoc(me.DynamicDocument):
    any_field = me.StringField()

import pandas as pd

all_csv_records = data_frame.to_dict('records')
data_frame = pd.read_csv(file_path)
for data in all_csv_records:
    report_data = DynamicDoc()
    DynamicDoc.any_field = str('temp_data')
    for col, row in data.iteritems():
        report_data[col] = row
    report_data.save()

Note - Only 'any_field' in mongo model is necessary input manullay in string form. Because field define manually in mongo model.

save id mongodb like:

How to do a Bulk Insert with MongoEngine?

2 Answers2