2

I am using Python as a driver to run thousands of numerical simulations. Since the initialization procedure for each simulation is the same, I can save time by doing the (time-consuming) initialisation procedure once, backup in memory the initial state of the vectors and for each subsequent simulation simply restore those vectors.

A small working example is:

from ctypes import cdll

try:
    lib = cdll.LoadLibrary('shared_lib.so')
    print lib.set_to_backup() # this tells the program to save the initial state
    print lib.simulate("cmd.txt") # initializes, backs up state, saves an internal variable -backed_up- to true and simulates
    print lib.simulate("cmd2.txt") # sees the internal state -backed_up- equal to true and skips initialisation, restores from memory and simulates
except:
    import traceback
    traceback.print_exc()

This works perfectly and I can run several simulations (cmd1, cmd2, ...) without reinitializing.

Now, I want to parallelize this procedure using multiprocessing. So, each process should load the library once and run the first simulation with initializing, saving and simulating. Each subsequent simulation should reload the initial state and simulate. The example for one process:

from ctypes import cdll
from multiprocessing import Process

try:
    lib = cdll.LoadLibrary('shared_lib.so')
    print lib.set_to_backup() # this tells the program to save the initial state
    p1 = Process(target=lib.simulate, args=("cmd.txt",)) # initializes, backs up state, saves an internal variable -backed_up- to true and simulates
    p1.start()
    p1.join()
    print p1.exitcode

    p2 = Process(target=lib.simulate, args=("cmd2.txt",)) # (should) see the internal state -backed_up- equal to true and skips initialisation, restores from memory and simulates
    p2.start()
    p2.join()
    print p2.exitcode
except:
    import traceback
    traceback.print_exc()

The first process does the job correctly (I can see it in the trace). The second process doesn't see the -backed_up- variable in lib and re-initialises everything.

I tried without declaring a new process, but simply reruning p1.start() to restart the same process but it fails (assert self._popen is None, 'cannot start a process twice').

-backed_up- is a global variable in lib and should remain in memory between calls to lib.simulate (as it does in the first example).

I run Linux Debian 7 and use python 2.7.3.

Anyone has an idea how to make this work please?

BenMorel
  • 34,448
  • 50
  • 182
  • 322
electrique
  • 431
  • 6
  • 18
  • I don't think you can do this with multiprocessing without using a shared memory buffer. Have you tried using multithreading instead. calling C functions bypasses the GIL so this should still give improved performance. – ebarr Mar 20 '14 at 10:47
  • @ebarr I could do it inside the C code with multithreading (ie OpenMP). But I'd like to keep the control on python so as to be able to check results, exit codes of each simulation, etc. Also, the C code would need a lot of restructuring to make it thread safe. While now only one simulation touches the data at each moment. – electrique Mar 20 '14 at 12:23
  • Thats why I am suggesting using the python mutlithreading package instead of multiprocessing package. If you have access to the C code, would it not be better to get it to return the initialisation state into a ctypes array that could be shared between processes? – ebarr Mar 20 '14 at 12:35
  • Have you tried `from threading import Thread as Process` and remove the corresponding `from multiprocessing import Process`? btw, move `p1.join()` after `p2.start()` – jfs Mar 21 '14 at 14:16

1 Answers1

1

I managed to get it to work using a queue. Answer heavily inspired by --> https://stackoverflow.com/a/6672593/801468

import multiprocessing
from ctypes import cdll

num_procs = 2

def worker():
    lib = cdll.LoadLibrary('shared_lib.so')
    print lib.set_to_backup()
    for DST in iter( q.get, None ):
        print 'treating: ', DST
        print lib.simulate(DST)
        q.task_done()
    q.task_done()

q = multiprocessing.JoinableQueue()
procs = []
for i in range(num_procs):
    procs.append( multiprocessing.Process(target=worker) )
    procs[-1].daemon = True
    procs[-1].start()

list_of_DST = ["cmd1.txt", "cmd2.txt"]
for DST in list_of_DST:
    q.put(DST)

q.join()

for p in procs:
    q.put( None )

q.join()

for p in procs:
    p.join()

print "Finished everything...."
print "Active children:", multiprocessing.active_children()
Community
  • 1
  • 1
electrique
  • 431
  • 6
  • 18
  • I don't see how it avoid reinitializing `num_procs` times. Unrelated: you could simplify the code using `multiprocessing.dummy.Pool(num_procs).map(lib.simulate, list_of_DST)` – jfs Mar 21 '14 at 14:21
  • @J.F.Sebastian Since the process keeps the same "memory" for the simulations it runs, the "avoiding" is internal to the shared library. After the first simulation, it internally sets a flag to true and each consequent simulation avoids the initialization. – electrique Mar 21 '14 at 20:57
  • yes. And it describes that each worker (all `num_procs` of them) initialize the library on its own. An alternative is to initialize the library in the parent before child processes are forked. – jfs Mar 22 '14 at 02:25
  • Have you tried `threading` solutions? (`.dummy` uses threads) – jfs Mar 22 '14 at 02:25
  • @J.F.Sebastian If I initialize in the main, will each worker take its own copy of the initialized library? Speed-wise, I don't think it'll make much difference. Let's assume initialization takes 3 sec and each simulation 7 sec and I have 8 simulations and 4 cpus. Case 1: each process initializes and solves in parallel -> 3+7+7=17 sec. Case 2: main initializes (3 s) and then each process solves in parallel (7+7). The same overall time in both cases. In the first comment you suggest, should I pass lib.simulate, or worker to map? – electrique Mar 22 '14 at 10:59
  • 1. a) each process initializes (4 processes, 8 simulations): `4*3 + 8*7/4 == 26` b) each process inherits: `3 + 8*7/4 == 17` c) `26 > 17` 2. You should pass `lib.simulate` as I wrote – jfs Mar 22 '14 at 11:20
  • @J.F.Sebastian I tried with map, but my shared library crashes when it tries to write something. On the other hand, initializing once in the main before passing down, works fine! Thanks! – electrique Mar 23 '14 at 13:44