Bulk create model objects in django

Question

I have a lot of objects to save in database, and so I want to create Model instances with that.

With django, I can create all the models instances, with MyModel(data), and then I want to save them all.

Currently, I have something like that:

for item in items:
    object = MyModel(name=item.name)
    object.save()

I'm wondering if I can save a list of objects directly, eg:

objects = []
for item in items:
    objects.append(MyModel(name=item.name))
objects.save_all()

How to save all the objects in one transaction?

It seems the ball is rolling on implementing a fix for this https://code.djangoproject.com/ticket/19527 — DanH, Jun 07 '13 at 06:55
wondering for list.save_all ? You could almost answer yourself just paraphrase that wondering and use 2 first words from your topic question. — Sławomir Lenart, Nov 02 '16 at 19:23

score 131 · Answer 1 · edited May 27 '20 at 06:25

131

as of the django development, there exists bulk_create as an object manager method which takes as input an array of objects created using the class constructor. check out django docs

edited May 27 '20 at 06:25

Amin Mir

640
8
15

answered Jan 16 '12 at 23:38

ecbtln

2,646
5
27
35

17

Django's docs for `bulk_create`: https://docs.djangoproject.com/en/dev/ref/models/querysets/#bulk-create – funkotron Jul 16 '12 at 15:37
3

But remember bulk_create has some limitations like it does not create primary keys if it is an AutoField which save() does automatically. – Hitesh Garg Aug 02 '15 at 14:58
1

@HiteshGarg, Is that still true now days? – Raydel Miranda Nov 22 '17 at 19:44
5

@RaydelMiranda, yes it is still true. It is right there in the documentation: `If the model’s primary key is an AutoField it does not retrieve and set the primary key attribute, as save() does, unless the database backend supports it (currently only PostgreSQL).` – interDist Apr 10 '18 at 17:04
How many objects should I bulk_create each time? Should I base on something (RAM, reality cases, etc.) to set the batch number? Let say I got to create billions records? – LucVH Dec 06 '18 at 07:13
2

Using **Django 3.0.x** and I confirm that using `bulk_create()` does not trigger any signals. I wonder why. – enchance Mar 12 '20 at 01:02
1

It is documented: "The model’s save() method will not be called, and the pre_save and post_save signals will not be sent." – Benedikt S. Vogler Mar 01 '21 at 11:43

score 77 · Accepted Answer · edited May 04 '19 at 00:42

77

Use bulk_create() method. It's standard in Django now.

Example:

Entry.objects.bulk_create([
    Entry(headline="Django 1.0 Released"),
    Entry(headline="Django 1.1 Announced"),
    Entry(headline="Breaking: Django is awesome")
])

edited May 04 '19 at 00:42

daaawx

3,273
2
17
16

answered Jan 26 '16 at 22:43

Danil

3,348
2
20
16

2

Changed in Django 1.10: Support for setting primary keys on objects created using bulk_create() when using PostgreSQL was added. – elad silver Jun 13 '17 at 14:46

score 7 · Answer 3 · answered Sep 04 '21 at 10:25

name = request.data.get('name')
period = request.data.get('period')
email = request.data.get('email')
prefix = request.data.get('prefix')
bulk_number = int(request.data.get('bulk_number'))

bulk_list = list()
for _ in range(bulk_number):
    code = code_prefix + uuid.uuid4().hex.upper()
    bulk_list.append(
        DjangoModel(name=name, code=code, period=period, user=email))

bulk_msj = DjangoModel.objects.bulk_create(bulk_list)

Thorin Schiffer · Answer 4 · 2021-09-04T08:27:00.530

6

worked for me to use manual transaction handling for the loop(postgres 9.1):

from django.db import transaction
with transaction.atomic():
    for item in items:
        MyModel.objects.create(name=item.name)

in fact it's not the same, as 'native' database bulk insert, but it allows you to avoid/descrease transport/orms operations/sql query analyse costs

edited Sep 04 '21 at 08:27

answered Nov 08 '12 at 13:39

Thorin Schiffer

2,818
4
25
34

2

This slightly changed. Now transaction does not have `commit_on_success` anymore. You should use `transaction.atomic()` See: https://stackoverflow.com/questions/21861207/is-transaction-atomic-same-as-transaction-commit-on-success – t_io Sep 14 '18 at 08:48
1

This method doesn't exactly scale, because you are practically making a create on every for loop pass which equals a db interaction per pass. bulk create seems to work faster. – inyang kpongette Sep 03 '21 at 04:48
Yeah this was answered a quite while ago, you should use bulk_create() now for this – Thorin Schiffer Sep 04 '21 at 08:27

score 4 · Answer 5 · answered Apr 29 '13 at 11:03

Here is how to bulk-create entities from column-separated file, leaving aside all unquoting and un-escaping routines:

SomeModel(Model):
    @classmethod
    def from_file(model, file_obj, headers, delimiter):
        model.objects.bulk_create([
            model(**dict(zip(headers, line.split(delimiter))))
            for line in file_obj],
            batch_size=None)

score 3 · Answer 6 · answered Sep 02 '10 at 23:01

Using create will cause one query per new item. If you want to reduce the number of INSERT queries, you'll need to use something else.

I've had some success using the Bulk Insert snippet, even though the snippet is quite old. Perhaps there are some changes required to get it working again.

http://djangosnippets.org/snippets/446/

score 2 · Answer 7 · answered Mar 19 '12 at 16:34

2

Check out this blog post on the bulkops module.

On my django 1.3 app, I have experienced significant speedup.

answered Mar 19 '12 at 16:34

MrJ

1,436
2
15
21

score -1 · Answer 8 · edited Mar 23 '16 at 09:59

-1

for a single line implementation, you can use a lambda expression in a map

map(lambda x:MyModel.objects.get_or_create(name=x), items)

Here, lambda matches each item in items list to x and create a Database record if necessary.

Lambda Documentation

edited Mar 23 '16 at 09:59

zEro

1,255
14
24

answered Aug 31 '10 at 12:14

Mp0int

18,172
15
83
114

1

You probably want to mention that the `lambda` has to be `map` ped over `items`: `map(lambda name: MyModel.objects.get_or_create(name = name), items)` – Manoj Govindan Aug 31 '10 at 16:24
Ja, thats another way i try to say (: – Mp0int Sep 01 '10 at 06:43

score -1 · Answer 9 · answered Sep 14 '22 at 16:28

-1

bulk_create() method is one of the ways to insert multiple records in the database table. How the bulk_create()

**

Event.objects.bulk_create([ Event(event_name="Event WF -001",event_type = "sensor_value"), Entry(event_name="Event WT -002", event_type = "geozone"), Entry(event_name="Event WD -001", event_type = "outage") ])

**

answered Sep 14 '22 at 16:28

Jitendra Meena

9
3

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Sep 19 '22 at 02:51

score -28 · Answer 10 · answered Aug 31 '10 at 11:45

-28

The easiest way is to use the create Manager method, which creates and saves the object in a single step.

for item in items:
    MyModel.objects.create(name=item.name)

answered Aug 31 '10 at 11:45

Daniel Roseman

588,541
66
880
895

+1. If `name` is unique and duplicate inputs are possible then it would be a good idea to use `get_or_create`. – Manoj Govindan Aug 31 '10 at 11:53
21

How does this answer the question? Model.objects.create is equivalent to object = MoModel(..) object.save(). And this does not do it in one transaction... – automagic Jan 31 '12 at 19:17

Bulk create model objects in django

10 Answers10

Linked

Related