Using requests library in on_failure or on_sucesss hook causes the task to retry indefinitely

Question

This is what I have:

import youtube_dl # in case this matters

class ErrorCatchingTask(Task):
    # Request = CustomRequest
    def on_failure(self, exc, task_id, args, kwargs, einfo):
        # If I comment this out, all is well
        r = requests.post(server + "/error_status/") 
        ....

@app.task(base=ErrorCatchingTask, bind=True, ignore_result=True, max_retires=1)
def process(self, param_1, param_2, param_3):
    ...
    raise IndexError
    ...

The worker will throw exception and then seemingly spawn a new task with a different task id Received task: process[{task_id} Here are a couple of things I've tried:

• Importing from celery.worker.request import Request and overriding on_failure and on_success functions there instead.
• app.conf.broker_transport_options = {'visibility_timeout': 99999999999}
• @app.task(base=ErrorCatchingTask, bind=True, ignore_result=True, max_retires=1)
• Turn off DEBUG mode
• Set logging to info
• Set CELERY_IGNORE_RESULT to false (Can I use Python requests with celery?)
• import requests as apicall to rule out namespace conflict
• Money patch requests Celery + Eventlet + non blocking requests
• Move ErrorCatchingTask into a separate file

If I don't use any of the hook functions, the worker will just throw the exception and stay idle until the next task is scheduled, which what I expect even when I use the hooks. Is this a bug? I searched through and through on github issues, but couldn't find the same problem. How do you debug a problem like this?

Django 1.11.16 celery 4.2.1

Answer 1

My problem was resolved after I used grequests

In my case, celery worker would reschedule as soon as conn.urlopen() was being called in requests/adapters.py. Another behavior I observed was if I had another worker from another project open in the same machine, sometimes infinite rescheduling would stop. This probably was some locking mechanism that was originally intended for other purpose kicking in.

So this led me to suspect that this is indeed threading issue and after researching whether requests library was thread safe, I found some people suggesting different things.. In theory, monkey patching should have a similar effect as using grequests, but it is not the same, so just use grequests or erequests library instead.

Celery Debugging instruction is here