I'm building a web application using Flask and I need to store a large amount of data that is organized as a string trie data structure. I'm struggling to figure out how to handle this data between requests, as the data is too large to be stored in memory.
Quick overview of the current process of my web application: when a request is made by a user, I retrieve a large serialized string of the trie data structure from S3 and then deserialize that said string to recreate a trie data structure. I save it as a global object so then when a user is clicking through the links shown on the page, a request is made to retrieve the children of the current node within the string trie data structure from that global object and then the data is shown as a link (if that node has a child) on the page and that process continues until there are no more children left. This current process works in development, but in production, using a global object of this data structure will not be viable.
Now I could store this data structure in some way to a database, but I would like to know an alternative approach to that. What are some ways to store this data between requests, so that it can be efficiently retrieved and used by the application? I'm open to using any libraries or tools that can help solve this problem.
Here's a high-level overview of the node object:
class Node:
def __init__(self, data, children):
self.data = data
# children stores other Node objects
self.children = {}
I appreciate any suggestions or insights that you may have. Thank you!
