what can i do to use dramatiq in flask app

Question

following this tutorial https://github.com/Bogdanp/dramatiq I created a simple python script to check if it works or not. I did some simple operations as follows

It did work fine.

please note that there are two terminals, and one to run script and one for worker, In the script given below print statements in count_word function gets executed in second terminal

working example
import pandas as pd
import dramatiq
import requests
import sys

@dramatiq.actor
def count_words(url):
    response = requests.get(url)
    count = len(response.text.split(" "))


    data = {'Name': ['Tom', 'nick', 'krish', 'jack'],
            'Age': [20, 21, 19, 18]}

    df = pd.DataFrame(data)

    df.to_csv(r'C:\Users\91956\Desktop\asdf.csv')
    print(df)
    print(f"There are {count} words at {url!r}.")


if __name__ == "__main__":
    count_words.send("http://whatsapp.com")

so I went ahead and tried to do the same thing in my flask app, i expected that the print statement in process_data will get printed in worker terminal, but it doesnt get printed at all, only "hi i am in work" gets printed in the terminal in which i run flask app

@app.route('/work')
def work():

    print("hi i am in work")    
    
    @dramatiq.actor
    def process_data():
        print('hi i am in process_data')
    
    process_data.send()
    
    return "hello"

I want to run process_data function in background, I am beginner and dont know much, is it possible to achieve it the way i am doing?

so I want to know how can I run specific function in background in flask?

score 0 · Answer 1 · answered May 28 '23 at 22:49

When process_data.send() executes, it enqueues a message in your broker, then returns to continue executing your function. Since your function immediately returns a response body, that HTTP request finishes immediately -likely before process_data has begun or has completed execution.

What can you do about this? First, accept that's just how it is with asynchronous or concurrent processing. Then, learn about the various strategies for retrieving the result of a delayed or async process.

Polling: periodically poll the server, by making HTTP requests that query for the result, until the result is available and retrievable
Establish a websocket between client and server, to receive requests and send results. This is probably the simplest solution that doesn't have the client polling the server, though initial coding is nontrivial, however good examples exist. E.g. see the Quart project, a Flask fork that implements a better websocket server solution: https://quart.palletsprojects.com/en/latest/
Employ a server "push" mechanism, where process_data tries to send the result upon completion, perhaps via a client omnibus query that returns either short notifications of any process completion results, or returns the results themselves
Make use of a framework that delivers server process results to the client automagically (e.g. RPC frameworks can do this, maybe React Redux does?)
Frameworks that implement the observer pattern, where a client callback function is invoked after a process on the server completes. Spring Boot provides this as a client-server observer pattern browser, not sure what's available for Python though.

I'd implement option #1 or #2, as they're the simplest. You can look around Pypi for the many RPC frameworks that do make a round-trip as callback, but I haven't found one that will do it for a browser client. I've built such a client-server RPC framework that runs over a websocket, between a Quart-based websocket app server and an EcmaScript browser client. One day I'll release it.

what can i do to use dramatiq in flask app

1 Answers1