16

All, I've got a issue with django signals.

I have a model In an effort to speed up responsiveness of page loads, I'm offloading some intensive processing that must be done, via a call to a second localhost webserver we're running, both using the same database. I'm seeing behavior where the calling process can retrieve the object, but the called process can't. Both port 80 and port [port] are pointing to django processes running off the same database.

In models.py

class A(models.Model):
    stuff...

def trigger_on_post_save( sender, instance, create, raw, **keywords):
    #This line works
    A.objects.get( pk=instance.pk )
    #then we call this
    urlopen( r'http://127.0.0.1:[port]' + 
        reverse(some_view_url, args(instance_pk) ).read()

post_save.connect( trigger_on_post_save, A )

In views.py

def some_view_function( request, a_pk ):
    #This line raises an object_not_found exception
    A.objects.get( pk=a_pk )

Furthermore, after the urlopen call raises an exception, the object does not exist in the database. It was my understanding that post_save was called after the object had been saved, and written to the database. Is this incorrect?

mklauber
  • 1,126
  • 10
  • 19

4 Answers4

17

We ran into a similar issue and we ended up using on_commit callback (NOTE: This is only possible with Django >= 1.9). So, you could possible do something like:

from django.db import transaction

class A(models.Model):
    stuff...

def trigger_on_post_save( sender, instance, create, raw, **keywords):
    def on_commit():
        urlopen(r'http://127.0.0.1:[port]' + 
                 reverse(some_view_url, args(instance_pk) ).read()
    transaction.on_commit(on_commit)

post_save.connect( trigger_on_post_save, A )

The idea here is that you wil be calling your endpoint after the transaction has been committed, so the instance involved in the transaction will be already saved ;).

Yoanis Gil
  • 3,022
  • 2
  • 15
  • 22
  • 1
    +1: This is the neat 'batteries included' solution for Django 1.9. Note that you can even use this from the 'pre_save' signal, since it will only fire when the transaction commits, so e.g. you can alter some model attributes as required before it ever hits the database.. – TimStaley Jul 11 '16 at 21:28
  • If you need to support Django 1.6 - 1.8 then look at https://django-transaction-hooks.readthedocs.io/en/latest/ which is the source of the feature that was added in 1.9 – Tim Tisdall Nov 24 '16 at 19:16
  • Be careful, this approach doesn't work with ModelSerializer.create from rest framework. – Mark Mishyn Jun 04 '18 at 06:55
  • @MarkMishyn did not know about that since I don't believe we were using DRF at the time. Would love to hear why it does not work with DRF. – Yoanis Gil Jun 05 '18 at 13:27
  • @YoanisGil because rest framework's ModelSerializer creates instances with M2M in 2 steps, so when `Model.objects.create` is called, M2M fields are empty even with `transaction.on_commit`. You can check here https://github.com/encode/django-rest-framework/blob/master/rest_framework/serializers.py#L940 – Mark Mishyn Jun 05 '18 at 18:07
14

I believe post_save fires after the save occurs, but before the transaction is commited to the database. By default, Django only commits changes to the database after the request has been completed.

Two possible solutions to your problem:

  1. Manage your transactions manually, and fire a custom signal after you commit.
  2. Have your second process wait a little while for the request to go through.

To be honest though, your whole setup seems a little bit nasty. You should probably look into Celery for asynchronous task queuing.

Evan Brumley
  • 2,468
  • 20
  • 13
  • +1 for the nod towards Celery. This is an ideal use-case for an async queue. – Steve Jalim Dec 16 '11 at 10:30
  • tbh, I'd love to get onto a Celery/RabbitMQ setup, but the previous developer already implemented his own TaskQueue system, so I was just trying to piggyback it. Thanks for the clarification about transactions, that's exactly what I was looking for. **EDIT**: I'll keep pushing our glacial software validation team to approve something like that. – mklauber Dec 16 '11 at 16:07
  • 9
    Just want to add that using celery doesn't change the issue - I've had a celery task getting old data because the transaction of the save() requested hadn't been committed yet. You either need to add a delay in your Task.run() or - better - instantiate the task "post_commit" rather than "post_save". Django doesn't provide this signal (yet) but have a look at https://github.com/davehughes/django-transaction-signals – Danny W. Adair Mar 13 '12 at 02:00
  • 5
    django's default behaviur is to commit on each save, not after request has been completed. Moreover, post_save signal is sent _after_ the commit – Ivan Virabyan Sep 28 '12 at 08:04
  • +1 for his setup seeming nasty and celery being a better option – Gattster May 14 '13 at 00:31
  • 2
    BTW, if you use Celery you need to call your task with a countdown because your object won't be available yet if you don't. Example, `tasks.mytask.apply_async(kwargs={'app_model': app_model, 'pk': instance.pk, 'field': 'photo'}, countdown=1)`. Or you could use, https://github.com/chrisdoble/django-celery-transactions. – Brent O'Connor May 17 '13 at 18:48
  • 1
    On 1.6, committed the transaction manually first before sending off to celery. Worked great so far. Example: `from django.db import transaction transaction.commit() tasks.mytask.delay(...)` – gdakram Apr 01 '14 at 21:59
  • Can't believe the comment by @IvanVirabyan isn't getting more upvotes, the first sentence of this answer is [patently false](https://github.com/django/django/blob/1.9.7/django/db/models/base.py#L733). – Dan LaManna Jul 09 '16 at 12:41
  • 1
    @gdakram Forcing a commit is a simple solution but can prematurely commit a transaction if the code you are operating in is actually wrapped in a larger transaction (e.g. saving multiple objects). Use with care. – JZC Oct 06 '16 at 18:06
  • @IvanVirabyan - `post_save` is not always called after the commit. If you're using `@atomic` on a view the `post_save` is called _before_ the commit. – Tim Tisdall Nov 24 '16 at 19:14
  • @TimTisdall yes, I said it wrong, I meant to say `post_save` is sent just after the database query is made. – Ivan Virabyan Nov 25 '16 at 10:26
  • I changed my complete solution using the accepted answer to find out the comment by @IvanVirabyan is the way how django handles signals and transaction. – Sumit May 28 '19 at 08:53
6

It's nice place to use decorators. There is slightly extended version of yoanis-gil's answer:

from django.db import transaction
from django.db.models.signals import post_save

def on_transaction_commit(func):
    def inner(*args, **kwargs):
        transaction.on_commit(lambda: func(*args, **kwargs))
    return inner


@receiver(post_save, sender=A)
@on_transaction_commit
def trigger_on_post_save(sender, **kwargs):
    # Do things here
raratiru
  • 8,748
  • 4
  • 73
  • 113
Mark Mishyn
  • 3,921
  • 2
  • 28
  • 30
  • This solution seems to cause an infinite loop. If I do an instance.save() in the trigger_on_post_save function, the .save() re-triggers the @on_transaction_commit, causing a recursive loop – Dev Jul 08 '20 at 18:01
  • @Dev you don't have to call save() in signals, cause it obviously leads to recusive calls. – Mark Mishyn Jul 10 '20 at 06:59
1

Had same issue when creating new model from django admin. Overriding ModelAdmin.save_model method to manage transaction manually worked.

def save_model(self, request, obj, form, change):
    from django.db import transaction
    with transaction.commit_on_success():
       super(ModelAdmin, self).save_model(request, obj, form, change)

    # write your code here
Venkat Kotra
  • 10,413
  • 3
  • 49
  • 53
  • `commit_on_success` was removed in Django > 1.8 see this for alternatives: https://stackoverflow.com/questions/21861207/is-transaction-atomic-same-as-transaction-commit-on-success – coler-j Sep 24 '18 at 23:59