Restart Flask App periodically to get most recent data and refresh multiple Python variables?

Question

I have a flask app that gets it's data from am oracle database using a SQL request. That database is updated very infrequently, so my idea was to get those data from the database and perform many manipulations of those data when the application loads. Then the web pages are very fast because I do not have make a SQL request again, and I also have no fear of SQL injection. However, I do need to update the data about once a day.

Below is a minimum verifiable example of the code which does not use SQL, but whichshould still work to demonstrate the principle.

In my real code df come from the database, but I create multiple variables from that that get passed into the web page using flask. How can I refresh these?

I tried this

How to re-run my python flask application every night?

but the server never refreshed, even when I put debug mode = False. The reason it doesn't work is that once the flask app is running it's like it's in it's own while loop serving the app. The first time I press ctrl+c it exits the app server and starts the while loop again, restarting the server, but that doesn't help.

from flask import Flask, render_template, redirect, url_for, Response
import pandas as pd
import datetime as dt  
import flask


#####################################################################
#start of part that I need to refresh

df= pd.DataFrame(range(5),columns=['Column1'])
df['Column2'] = df['Column1']*2
df['Column3'] =dt.datetime.now()
df['Column4'] = df['Column3'] + pd.to_timedelta(df['Column2'],'d')

var1=min(df['Column4'])
var2=max(df['Column4'])

#End of Refresh
####################################################################

app = flask.Flask(__name__, static_url_path='',
            static_folder='static',
            template_folder='template')
app.config["DEBUG"] = True

@app.route("/home")
def home():
return render_template("home.html",
                       var1=str(var1),
                       var2=str(var2)) 
if __name__ == '__main__':
    app.run(host="localhost", port=5000, debug=True)

HOME HTML

'''
<HTML>
    <body>
        <div>This is Var1: {{var1}} </div>
        <div>This is Var2: {{var2}} </div>
    
    </body>
</HTML>
'''

I would like to make a working example that refreshed every 5 minutes as a proof of concept.

Schedule a repeating task like this: https://stackoverflow.com/questions/2398661/schedule-a-repeating-event-in-python-3 — Daniel Scott, Nov 17 '21 at 23:27
I need to assign a bunch of variables in a namespace where I can pass them to my web page. THat is on eplace I run into an issue with this. — bart cubrich, Nov 17 '21 at 23:54
It seems to me the main issue is that once I get into app.run() the scheduler isn't being run anymore. I can get the scheduler to work when it does not involve a flask app. — bart cubrich, Nov 18 '21 at 00:55

score 1 · Accepted Answer · answered Nov 19 '21 at 18:19

Based on the code you provided, the part you want to refresh will only be run once in the startup of your main process. You could have a cron job outside of your flask server that restarts it but this will cause a downtime if someone tries to query while the server is being restarted.

A better solution to this, is to add this query and the manipulation of the data in a function and call it every time someone tries to access the page. This way, you can set a cache which will query the data once and it will save them in memory. You also have the option to specify how long you want the data to be cached, and it will automatically drop the data until the next time someone asks for them.

from cachetools import TTLCache

# Cache the results with a time to live cache. A cached entry will get deleted
# after 300s 
TTLCache(maxsize=200, ttl=300)
def get_data():
    
    #####################################################################
    #start of part that I need to refresh

    df= pd.DataFrame(range(5),columns=['Column1'])
    df['Column2'] = df['Column1']*2
    df['Column3'] =dt.datetime.now()
    df['Column4'] = df['Column3'] + pd.to_timedelta(df['Column2'],'d')

    var1=min(df['Column4'])
    var2=max(df['Column4'])

    #End of Refresh
    ####################################################################
    return var1, var2

Now you can call this function but it will only get executed if the result is not already saved in cache.

@app.route("/home")
def home():
    var1, var2 = get_data()
    return render_template("home.html",
                   var1=str(var1),
                   var2=str(var2))

Another advantage of this solution is that you can always clear the cache completely when you have updated the data.

This is great. Another solution which I found was to save to a local .db file and have a chron job elsewhere that access the Oracle db with the data I need, then ssh that data to my server. This is actually necessary in one of my uses cases, as my server is not within the firewall of the other server, but in my other use case this will be a life saver. — bart cubrich, Nov 19 '21 at 18:31
In order to work in my specific use case UI had to use `cacheout` instead of cache tools because I could not cache a pandas df correctly. This answer answered my original question with the example I used. — bart cubrich, Nov 19 '21 at 23:14

Restart Flask App periodically to get most recent data and refresh multiple Python variables?

1 Answers1