django get_or_create - performance optimization for a list of objects

Question

Consider the following (pseudoPython) code:

l = [some, list]
for i in l:
    o, c = Model.objects.get_or_create(par1=i["something"], defaults={'par2': i["else"],})

assuming that most of the time the objects would be retrieved, not created,

there is an obvious performance gain by quering with a first SELECT() of objects not in the set defined by par1, and then bulk-inserting the missing ones..

but, is there a neat Python/Django pattern of accomplishing that without diving into SQL?

This is a bulk import routine, so l contains dictionaries, not django model instances.

possible duplicate of [Efficent way to bulk insert with get\_or\_create() in Django (SQL, Python, Django)](http://stackoverflow.com/questions/2252530/efficent-way-to-bulk-insert-with-get-or-create-in-django-sql-python-django) — KillianDS, Jul 01 '14 at 06:44

score 1 · Answer 1 · answered Jul 01 '14 at 05:12

1

Given a list of IDs, you can use Django to quickly give you the corresponding Model instances using the __in operator: https://docs.djangoproject.com/en/dev/ref/models/querysets/#in

photos_exist = Photo.objects.filter(
            id__in=photo_ids
            )

answered Jul 01 '14 at 05:12

johntellsall

14,394
4
46
40

Yeah, but what about the case where there are two parameters, both of which need to match? – qdot Jul 01 '14 at 05:21
Nope. In that case I'd do a couple variations and see which is easiest; your original idea. If you find a shortcut let me know! – johntellsall Jul 01 '14 at 05:31

score 1 · Answer 2 · answered Jul 01 '14 at 06:50

You can use Q objects to create a complex query to SELECT the existing rows. Something like:

query_parameters = Q()
for i in l:
    query_parameters |= Q(first=i['this']) & Q(second=i['that'])

found = MyModel.objects.filter(query_parameters)

Then you can figure out (in Python) the rows that are missing and create() them (or bulk_create() for efficiency, or get_or_create() if there are potential race conditions).

Of course, long complex queries can have performance problems of their own, but I imagine this would be faster that doing a separate query for each item.

django get_or_create - performance optimization for a list of objects

2 Answers2