0

I'm looking for the best way to keep big objects in memory in a flask application, such that I can access it (read-only) inside every request without storing it on disk or in any database. These objects are not serializable since they are complex instances of classes from imported libraries (then an in-RAM database is not an option). My major problem is that flask uses multiple processes, so defining a global variable may result in several copies of the same objects (which I want to avoid). My main idea to tackle this is the following:

  • when the flask application is started, I create a unique subprocess which will load the data from disk and keep it in the RAM (as regular python objects).
  • inside a request, I get the data by communicating with the subprocess.

Do you know a library that already does this ? Do you have a better idea to access these data as fast as possible ?

Robin
  • 1,531
  • 1
  • 15
  • 35
  • 3
    Data needs to be serializable in order to be sent from one process to another – zvone Sep 05 '18 at 08:41
  • 1
    What you are planning to do might work well (to some degree at least) on the Flask development server. It will definitely not work well in a production setup of Flask where a WSGI server will handle process and thread management as you have experienced. There is not library to magically solve that. The options to share data between processes are kind of limited. So the best way would be to instruct the WSGI server to run only one process. And don't open extra processes inside, that can have side effects on most WSGI servers. – Klaus D. Sep 05 '18 at 08:53
  • @zvone you right... But I guess the only way to do what I'm trying is to find a way to serialize the data, and use Redis in memory. – Robin Sep 05 '18 at 10:12

0 Answers0