Django bulk_create using a list of random fields without a hardcoded query

Question

I am trying to create new instances in my Django model using bulk_create().

My models are:

Class Emotion(models.Model):
    emotion_grp_id = models.AutoField(primary_key=True, ...
    emotion_group = models.CharField(max_length=55, ...

Class EmotionSubGroup(models.Model):
    emotion_sub_grp_id = models.AutoField(primary_key=True, ...
    emotion_grp = models.ForeignKey(EmotionGroup, on_delete=models.CASCADE, ...
    emotion_sub_group = models.CharField(max_length=55, ...

views.py

The relevant portion of the function that I am using looks like this:

def DataUpload (request):
   # ...
   # ...

    df_rawdata = pd.read_csv(csv_file, sep='').   # Dataframe from source (.csv) file

    row_iter = df_rawdata.iterrows()
    data = [
        EmotionSubGroup(
            emotion_grp_id=row['emotion_grp'],
            emotion_sub_group=row['emotion_sub_group'],
            )
        for index, row in row_iter
        ]         

    EmotionSubGroup.objects.bulk_create(data)

Is it possible to create a general data structure instead of hardcoding the field names, for example:

data = `list_of_target_fields = rows_from_external_source`

to be equivalent of what I am using currently and carry out the upload.

Getting error `Cannot assign "1": "EmotionSubGroup.emotion_grp" must be a "EmotionGroup" instance.` — carla, Dec 13 '22 at 05:13
`emotion_grp_id` is it **ForeignKey** ? If yes then you've to specify `emotion_grp` instance instead of `id` — Ankit Tiwari, Dec 13 '22 at 05:18
If you take a look at my query above, using `emotion_grp_id=row['emotion_grp']`, I am able to run the function. If I change it to `emotion_grp=row['emotion_grp']` I am running into the same error as cited above `Cannot assign "1": "EmotionSubGroup.emotion_grp" must be a "EmotionGroup" instance.`. And this is what I am getting if I use the **modified query** with `EmotionSubGroup(**row)`. I think it makes sense too as model `EmotionSubGroup` is related to the parent `Emotion` of which `emotion_grp_id` is the primary key? — carla, Dec 13 '22 at 05:29
So exception is **ForeignKey** then update your `row` with foreignkey field eg. `row['emotion_grp_id'] = row.pop('emotion_grp')` — Ankit Tiwari, Dec 13 '22 at 05:53
So how do I translate this to your original solution `EmotionSubGroup(**row)`? My goal is to ultimately get rid of the field mapping with a **general query statement**. — carla, Dec 13 '22 at 05:57
You can create a function which can do this for you. I'll add a function for you in answer — Ankit Tiwari, Dec 13 '22 at 05:58

Ankit Tiwari · Accepted Answer · 2022-12-13T07:09:03.460

0

Here is a simple function which will update ForeignKey and will return a instance of model

def set_instance(data, model, foreignkeys=None):
    if foreignkeys:
       # update foreignkey in data dict
       for k, v in foreignkeys.items():
          data[v] = data.pop(k)
    return model(**data)
  

def DataUpload (request):
    # ...
    # ...

    df_rawdata = pd.read_csv(csv_file, sep='').   # Dataframe from source (.csv) file

    row_iter = df_rawdata.iterrows()
    foreignkey_fields = {'emotion_grp':'emotion_grp_id'}
    data = [
        set_instance(row, EmotionSubGroup, foreignkey_fields)
        for index, row in row_iter
        ]         

    EmotionSubGroup.objects.bulk_create(data)

Other approch to make code readable

def update_foreignkey(data, foreignkeys=None):
    if foreignkeys:
       # update foreignkey in data dict
       for k, v in foreignkeys.items():
         data[v] = data.pop(k)
    return data
  

def DataUpload (request):
    # ...
    # ...

    df_rawdata = pd.read_csv(csv_file, sep='').   # Dataframe from source (.csv) file

    row_iter = df_rawdata.iterrows()
    foreignkey_fields = {'emotion_grp':'emotion_grp_id'}
    data = [
        EmotionSubGroup(**update_foreignkey(row, foreignkey_fields))
        for index, row in row_iter
        ]         

    EmotionSubGroup.objects.bulk_create(data)

edited Dec 13 '22 at 07:09

answered Dec 13 '22 at 06:09

Ankit Tiwari

4,438
4
14
41

I have a query: What if I use the first solution i.e. with the `model` thrown in?Getting your drift, I have used the second solution and it works. How will the first solution make the query different? ... and supposing I have a model (there are, many) with **multiple foreign keys**. So in those cases I'll need to construct the `foreignkey_fields = ` dict? as in: `{'emotion_grp':'emotion_grp_id', 'child_key1':'f_key1'}, like so? – carla Dec 13 '22 at 06:50
First solution is not readable becouse it conatain a method which does every thing and in second solution is more readable you can see that your model accepting updated fields from a method. – Ankit Tiwari Dec 13 '22 at 07:05
I've created `foreignkey_fields` to change your `emotion_grp` filed with `emotion_grp_id` check I've updated my answer if you don't have any foreignkey then you can skeep that – Ankit Tiwari Dec 13 '22 at 07:07
Sorry I think I could not convey my question properly. What I am pointing at, suppose in the child model **EmotionSubGroup** I have another field `another_key` with FK relationship to its parent through its primary key `another_key_id`. In that case I would need to modify my dict of FK fields to : `foreignkey_fields = {'emotion_grp':'emotion_grp_id', 'another_key':'another_key_id'}`. Is that correct? – carla Dec 13 '22 at 07:19
Hello @carla Yes, if you want to specify diffrent filed `another_key_id` instead of default field `another_key` then you can add those fileds inside `foreignkey_fields` key will be original filed `another_key` and value will be which field you want to replace `another_key_id` – Ankit Tiwari Dec 13 '22 at 07:28

Django bulk_create using a list of random fields without a hardcoded query

views.py

1 Answers1