Celery tasks run twice in the background once Flask application is run, why?

Question

I'm building a Flask web application, where I want some asynchronous background tasks to be started once the Flask application is run. I've been reading about the examples and manuals regarding Flask and Celery, e.g. here but in many examples the Celery background tasks are bound to the Flask routing functions. That is, the Celery background functions are run when a request for a specific URL-bound Flask function is received from the client.

What I want, in addition to this, is to have Celery background tasks started the moment the Flask application is run. I have created a minimal working example into this post (myapp.py, Flask + Celery + RabbitMQ):

from flask import Flask
from celery import Celery
import time

app = Flask(__name__)

celery = Celery(app.name, backend='rpc://', broker='pyamqp://')

@celery.task
def long_background_task1():
    print("Doing the heavy background job 1...")
    time.sleep(5)
    print("Finished the heavy background job 1!")
    return("This is the return from the heavy background job 1!")

@celery.task
def long_background_task2():
    print("Doing the heavy background job 2...")
    time.sleep(10)
    print("Finished the heavy background job 2!")
    return("This is the return from the heavy background job 2!")


@app.route("/")
def index():
    return("This is the index page")


if __name__ == '__main__':
    # At the start of the Flask application, start also one or more background Celery tasks
    long_background_task1.delay()
    long_background_task2.delay()
    app.run(debug=True)

So when myapp.py is run, I want the Celery tasks long_background_task1 and long_background_task2 to be started and run on the background. If I start the Celery worker in the terminal with the command:

celery -A myapp.celery worker --loglevel=INFO

I get the following output:

So Celery seems to be okay! Next, when I run myapp.py I get the following output in Celery terminal (click the image to enlarge):

So the Celery tasks are indeed started in the background, but they are run two times! What causes this?

So my question is: How can make the Celery background tasks start on the background only once?

score 1 · Accepted Answer · answered May 23 '21 at 21:00

I think I found the solution myself. I was checking at this post:

https://www.reddit.com/r/flask/comments/2chlio/does_init_get_run_twice_in_my_case/

And I noticed the comment:

"The development server does all sorts of things that won't happen in production to make sure that changes to files are loaded into the application live."

So I changed in myapp.py:

app.run(debug=True)

to

app.run(debug=False)

and the problem was fixed! No running the tasks twice anymore.

Celery tasks run twice in the background once Flask application is run, why?

1 Answers1