Make a Celery task that waits for a signal?

Asked Nov 19 '13 at 02:15

Active Nov 19 '13 at 02:20

Viewed 532 times

Is it possible to create Celery task that just waits for a signal? I have this scenario:

Scrapyd in one virtualenv on remote machine A
Django project with Celery worker node in another virtualenv on remote machine A
The same Django project with Celery, but in another virtualenv on local machine B

How I use this setup:

I would send a task chain from Django on machine B
Let the task chain be consumed by the worker node on machine A.
In the first subtask of the task chain, I would schedule a crawl using Scrapyd's JSON over HTTP API, and pass the Celery task ID to the crawler as an HTTP request parameter.
I then want this first subtask to just wait for some kind of signal.
Scrapyd does its thing and runs the spider.
Once the spider is done crawling, I want it to send a signal, maybe by JSON over HTTP or by a Django management command, to the subtask that has been waiting for the signal.

Is this doable?

I would just need code snippets to show me how to wait for a signal in a subtask, and how to restore a task from the task ID and send a signal to it.

edited Nov 19 '13 at 02:20

asked Nov 19 '13 at 02:15

Kal

Waiting for something in worker is really bad idea. Try rearranging the logic forking a subprocess which handles long operations and then queues another task to complete the operation. – Vitaly Greck Apr 06 '15 at 23:56
Break this process in 3 tasks. – Nikolay Fominyh Mar 14 '16 at 11:22

0 Answers0