3

I have a very large file that loads in my main process. my goal is to have several processes read from memory at the same time to avoid memory constraints and to make it faster.

according to this answer, I should use Shared ctypes Objects

Manager types are built for flexibility not efficiency ... this necessarily means copying whatever object is in question. .... If you want shared physical memory, I suggest using Shared ctypes Objects. These actually do point to a common location in memory, and therefore are much faster, and resource-light.

so I did this:

import time
import pickle
import multiprocessing
from functools import partial

def foo(_, v):
    tp = time.time()
    v = v.value
    print(hex(id(v)))
    print(f'took me {time.time()-tp} in process')

if __name__ == '__main__':
    # creates a file which is about 800 MB
    with open('foo.pkl', 'wb') as file:
        pickle.dump('aaabbbaa'*int(1e8), file, protocol=pickle.HIGHEST_PROTOCOL)

    t1 = time.time()
    with open('foo.pkl', 'rb') as file:
        contract_conversion = pickle.load(file)
    print(f'load took {time.time()-t1}')

    m = multiprocessing.Manager()
    vm = m.Value(str, contract_conversion, lock=False)  # not locked because i only read from it so its safe
    foo_p = partial(foo, v=vm)

    tpo = time.time()
    with multiprocessing.Pool() as pool:
       pool.map(foo_p, range(4))
    print(f'took me {time.time()-tpo} for pool stuff')

however I can see that the processes use a of copy it (the ram in each process is very high) and it's MUCH slower than simply reading from disk.

the print:

load took 0.8662333488464355
0x1c736ca0040
took me 2.286606550216675 in process
0x15cc0404040
took me 3.178203582763672 in process
0x1f30f049040
took me 4.179721355438232 in process
0x21d2c8cc040
took me 4.913192510604858 in process
took me 5.251579999923706 for pool stuff

also the id is not the same, though I am not sure if id is simply a python identifier or the memory location.

moshevi
  • 4,999
  • 5
  • 33
  • 50
  • Rather than reading the whole file into shared memory, is it possible for each of your sub-processes to read just their chunk of the file from disk? It depends on the type of data that's in the file and if you know what segments to read ahead of time. – Matt S Oct 02 '18 at 13:53
  • not really, all the processes use the same chunks, but in a different way. its Not shown here but I pass additional parameters to the processes. – moshevi Oct 02 '18 at 14:06
  • If you load too much into memory, your PC might save some of that memory on disk (this is called "swapping"), and it is heavily reducing the performance of an application. Might this be the case here? – Dominique Oct 02 '18 at 14:46
  • I have about 16 GB ram on my laptop even if i create the file with `'aaabbbaa' * int(1e7)` it takes about three times as long to retrieve the shared data then from file. the memory usage does not exceed 40% – moshevi Oct 02 '18 at 14:57

1 Answers1

4

You're not using shared memory. That would be multiprocessing.Value, not multiprocessing.Manager().Value. You're storing the string in the manager's server process and sending pickles over TLS connections to access the value. Also, the server process is limited by its own GIL when serving requests.

I don't know how much each of those aspects contributes to the overhead, but it's overall more expensive than reading shared memory.

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • I tried to pass a binary object using `ctypes.c_char_p` but i said `c_char_p objects should only be shared between processes through inheritance`. – moshevi Oct 03 '18 at 06:32
  • @moshevi: Yeah, I'm not sure if there's any better way to share a `multiprocessing.Value` with child processes than going through a global. I don't see any way to make it work through `args`. – user2357112 Oct 03 '18 at 06:56
  • what do you mean by "going through a global"? I was no aware of any other way to pass data to a child process apart from args. – moshevi Oct 03 '18 at 07:00
  • Apparently it might be possible to pass the Value to the workers by giving the pool an `initializer` and `initargs`. I don't have time to test it out right now. – user2357112 Oct 03 '18 at 07:16