I want to share a large read only object between subprocess made by multiprocessing library on Linux. I don't want to use Manager() because that affects performance.
The large object is not an array and is actually a list of gmpy.xmpz from the gmpy2 library. This means using a shared array from the multiprocessing library doesn't work.
How to do that?
In the past, when I used python 2, I can use a global variable in the main process. The subprocess can read it without making copies of the object on linux, as long as I don't modify the object in any way.
But, somehow the trick doesn't work anymore after I recompile python 2 with intel compiler. I also migrate my code to python 3. The same trick still doesn't work.
old code:
import multiprocessing as mp
import numpy as np
def f(x):
n = 0
print(shared_obj[0], id(shared_obj))
def main():
global shared_obj
# 7Gb object. in the actual code, this is not an array
shared_obj = np.zeros(7e9/4, dtype=np.int32)
pool = mp.Pool(6)
result = pool.map_async(f, range(10))
pool.close()
result.get()
pool.join()
main()
Error:
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
