Django: IntegrityError (Duplicate Entry) fixed, now getting DoesNotExist

Question

I've been having a rought time saving data to a MySQL database.

This is on a Django 1.4.22 legacy project running several celery-driven tasks to generate reports.

At first I used the get_or_create method but soon it was clear that it wasn't operating correctly as they were a lot of Duplicate Entry errors. So I relied to stuff like the following, making a queryset, then checking it with the exists() method, then creating it if necessary or updating it:

There's only one instance to be found each time (that's why I want to make a update or create action)

qs_snapshot = StatsSnapshot.objects.filter(list=list_id,
                                           created_at__gte=today,
                                           created_at__lt=end_day)
if qs_snapshot.exists():
    snapshot = qs_snapshot.get()
    qs_snapshot.update(**dict_stats)
else:
    try:
        snapshot = StatsSnapshot.objects.create(**dict_stats)
    except IntegrityError:
        # Duplicate entry error, upgrade the data instead or recreating it
        # There's no atomicity guaranted so sometimes there's really an
        # instance saved after the exists() and before the create()
        snapshot = qs_snapshot.get()
        qs_snapshot.update(**dict_stats)

So I've got this ugly try, but still it doesn't do what i'm specting as on the exception body, the snapshot = qs_snapshot.get() now raises a DoesNotExist: StatsSnapshot matching query does not exist. How's this even possible? I'm double checking!

I've tried several tips given here, including the fix for the mysql sequences but nothing seems to fix it. Also it does happen with different reports (lists) each time

Bogdan Goie · Answer 1 · 2016-01-25T17:21:55.783

0

Update:

I read through the code again and noticed an issue: inside the except IntegrityError:, you can't do qs_snapshot.get() because there are no objects returned by qs_snapshot. You checked before qs_snapshot.exists() and if not, you can't do .get() on the queryset because there is no object, that's why you get the DoesNotExist exception.

Update 2: Based on your comments try the following:

...
else:
    try:
        snapshot = StatsSnapshot.objects.get(**dict_stats)
    except StatsSnapshot.DoesNotExist:
        snapshot = StatsSnapshot.objects.create(**dict_stats)
    except StatsSnapshot.MultipleObjectsReturned:
        # it returned multiple objects, decide here what you want to do
        pass

edited Jan 25 '16 at 17:21

answered Jan 25 '16 at 11:53

Bogdan Goie

1,257
1
13
21

I've updated the original question clarifying that at query time there's either none or 1 instance created for the given list and created_at arguments. That's why there's no MultipleObjectsReturned exception catched – Javi Romero Jan 25 '16 at 16:23
Also, the qs_snapshot.get() without arguments is not the root of the problem as it returns the element on the queryset if there's only one: qs_snapshot.get() == qs_snapshot[0] if qs_snapshot.count() == 1 ``` >>> today = timezone.now().replace(hour=0, minute=0, second=0, microsecond=1) >>> end_day = today + datetime.timedelta(1) >>> qs = StatsSnapshot.objects.filter(list=1413, created_at__gte=today, created_at__lt=end_day) >>> qs.count() 1 >>> snap = qs.get() >>> snap.pk 24933L >>> snapindex = qs[0] >>> snapindex.pk 24933L ``` – Javi Romero Jan 25 '16 at 16:31
@Gooler read again your code and noticed where the problem actually lies. I updated the answer. – Bogdan Goie Jan 25 '16 at 16:43
Yeah, that makes sense to me too, but the point is that the code enters the except IntegrityError clause because there's actually another object on the DB at .create() time. I'll try adding again the queryset statement just in case it's cached somehow and needs reevaluation... – Javi Romero Jan 25 '16 at 16:52
@Gooler Try to change to StatsSnapshot.objects.get_or_create(**dict_stats) instead of just create, assuming there is only one object in databse that contains the values in dict_stats if it exists – Bogdan Goie Jan 25 '16 at 16:58
Yeah, I've already tried that before this if-else-try-boom abomination and i was facing the same error, IntegrityError (duplicated entry...) get_or_create is not waranteed to be atomic http://stackoverflow.com/questions/6416213/is-get-or-create-thread-safe – Javi Romero Jan 25 '16 at 17:06
@Gooler ok then , try to get or create manually then. I updated the answer with code in Update 2. – Bogdan Goie Jan 25 '16 at 17:22

Django: IntegrityError (Duplicate Entry) fixed, now getting DoesNotExist

1 Answers1