Imagine I have multiple python processes running from independent scripts on a single machine. They all need read-only access to a monolithic data object, say a pandas.DataFrame
. Assume that the object is so large that only one copy will fit into memory. I would like to implement something similar to this question but with more of a local server architecture without spawning all my workers from a single head process via multiprocessing
. If the object was fully immutable I would use Plasma, but unfortunately the server may have to change the contents from time to time.
Can you recommend a good python package or some example code to do that?