Python - Returning query result from server to client in Flask

Question

• What I have

  I have a Client/Server in Flask. The client sends a query in JSON format to the server and the server creates a JSON file. There is another tool which takes this query, executes it on a db and writes the result to a results.txt file. The server periodically checks the 'results' directory for .txt files and if it finds a new file it extracts the result. For the periodic checking part I used APS.

• What I want to do Now I want to send this data (queryResult) which the server has extracted from the .txt file back to the client.

This is what I have done so far.

• Server Code:

app = Flask(__name__)
api = Api(app)

# Variable to store the result file count in the Tool directory
fileCount = 0

# Variable to store the query result generated by the Tool
queryResult = 0    
    
# Method to read .txt files generated by the Tool
def readFile():
    global fileCount
    global queryResult
    # Path where .txt files are created by the Tool
    path = "<path>"
    tempFileCount = len(fnmatch.filter(os.listdir(path), '*.txt'))
    if (fileCount != tempFileCount):
        fileCount = tempFileCount
        list_of_files = glob.iglob(path + '*.txt')
        latest_file = max(list_of_files, key=os.path.getctime)
        print("\nLast modified file: " + latest_file)
        with open(latest_file, "r") as myfile:
            queryResult = myfile.readlines()
            print(queryResult) # I would like to return this queryResult to the client
            
scheduler = BackgroundScheduler()
scheduler.add_job(func=readFile, trigger="interval", seconds=10)
scheduler.start()   

# Shut down the scheduler when exiting the app
atexit.register(lambda: scheduler.shutdown())   
   
# Method to write url parameters in JSON to a file
def write_file(response):
    time_stamp = str(time.strftime("%Y-%m-%d_%H-%M-%S"))  
    with open('data' + time_stamp + '.json', 'w') as outfile:
        json.dump(response, outfile)
    print("JSON File created!")
   

class GetParams(Resource):
    def get(self):
        response = json.loads(list(dict(request.args).keys())[0])  
        write_file(response)  
           
api.add_resource(GetParams, '/data')  # Route for GetJSON()

if __name__ == '__main__':
    app.run(port='5890', threaded=True)

• Client Code

data = {
    'query': 'SELECT * FROM table_name'
}

url = 'http://127.0.0.1:5890/data'

session = requests.Session()
retry = Retry(connect=3, backoff_factor=0.5)  
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

resp = session.get(url, params=json.dumps(data))
print(resp)

Can anyone please help me as to how to send this queryResult back to the Client?

EDIT: I would like the server to send the queryResult back to the client each time it encounters a new file in the Tool directory i.e., every time it finds a new file, it extracts the result (which it is doing currently) and send it back to the client.

Answer 1

What you want to do is called a Web Worker Architecture.

To pass in a real-time abstract queryResult from a background job to a client app you can use a combination of a Message Queue (Kafka is recommended, RabbitMQ is OK too) and a Web-Socket. When a client sends a request to the /data endpoint, you should return back some unique token (like UUID if your user is anonymous, or user id if it's authenticated). The same token you should add to the name of the resulting file. When your background worker finished processing the file it uses the token (from a file's name) to create a Kafka or RabbitMQ topic, like topic_for_user_id_1337 or topic_for_uuid_jqwfoj-123qwr, and publishes queryResult as a message.

At the same time, your client should establish a web-socket connection (Flask is quite bad for web-sockets, but there are few fine libs to do that anyway, like socketio) and pass the token through it to your backend so it'll create a message queue subscriber, subscribed to a topic with the token's name so when a background job is finished, a web-backend will receive a message and pass it to the user through a web-socket.

P.S. If it sounds overly complicated, you can avoid the use of MQ and WS and put the queryResult into a db and create a endpoint to check if it exists in a DB. If it's not, you return something like not ready yet and a client retries in a few seconds, if it's ready -- you return the queryResult from the DB.