Notify or callback flask after remote celery worker is completed

Question

I am running celery client(Flask) and worker in two different machines, now once the worker has completed the task, I need to callback a function on client side. Is this possible?

Celery client:-

celery_app=Celery('test_multihost', broker='amqp://test:test@<worker_ip>/test_host', backend='rpc')
result= testMethod1.apply_async((param1, param2,param3), link=testMethod2.s())

@celery_app.task
def testMethod2():
    #testMethod2 body.

Celery Worker:-

celery_app=Celery('test_multihost', broker='amqp://test:test@<worker_ip>/test_host', backend='rpc')
@celery_app.task
def testMethod1():
   #testMethod1 body

But the problem is the function testMethod2 is getting executed on the celery worker side, not on the client side.

Is there anyway that I can callback the method on client side?

Ruben Helsloot · Answer 1 · 2020-08-14T13:35:32.840

0

One way to do this is to have Celery write its result in a database table, and use Flask to poll for the result of the task by repeatedly querying the database. A similar construct might be to keep a register of completed tasks in Redis, but the gist would be the same.

Do you want to trigger a completion message to the user? If you can notify by email/text message, you could just let Celery handle that of course.

If you need to kickstart some Flask process - and it really needs to be inside Flask's ecosystem for some reason - use the worker with the requests module to call to an endpoint that Flask is listening to.

edited Aug 14 '20 at 13:35

answered Aug 14 '20 at 13:08

Ruben Helsloot

12,582
6
26
49

polling would my last option. If it was email I would send it through celery worker but what I want is to call a method for some data processing within Flask application after celery worker completes its task. – sattva_venu Aug 14 '20 at 13:29
1

But why do you need that to be done *inside* flask? Remember that it's a web application, and what you're describing is not what it was built for. If you really need to do so, use the worker with `requests` to call to an endpoint that Flask is listening to. That way, you can kick off your data processing step. – Ruben Helsloot Aug 14 '20 at 13:34
I am new to Flask and Celery. Thanks for your suggestion. Will give a try on this. – sattva_venu Aug 14 '20 at 14:50
I found the solution using celery signals – sattva_venu Aug 18 '20 at 06:53
Nice! Could you add it below for future reference? – Ruben Helsloot Aug 18 '20 at 07:02
I have added it. – sattva_venu Aug 18 '20 at 10:51

score 0 · Accepted Answer · answered Aug 18 '20 at 10:49

I solved this problem using @after_task_publish from celery signals. The code snippet is as follows:-

@after_task_publish.connect(sender=<registered_celery_task>)
def testMethod2(sender=None, headers=None, body=None, **kwargs):
    #callback body

The testMethod2 will be called after the celery worker is completed on the remote machine. Here I can access the result of celery worker using headers parameter.

Notify or callback flask after remote celery worker is completed

2 Answers2