Multiprocessing takes longer?

Question

I am trying to understand how to get children to write to a parent's variables. Maybe I'm doing something wrong here, but I would have imagined that multiprocessing would have taken a fraction of the time that it is actually taking:

import multiprocessing, time

def h(x):
    h.q.put('Doing: ' + str(x))
    return x

def f_init(q):
    h.q = q

def main():
    q = multiprocessing.Queue()
    p = multiprocessing.Pool(None, f_init, [q])
    results = p.imap(h, range(1,5))
    p.close()

-----Results-----:

1
2
3
4
Multiprocessed: 0.0695610046387 seconds
1
2
3
4
Normal: 2.78949737549e-05 seconds # much shorter

    for i in range(len(range(1,5))):
        print results.next() # prints 1, 4, 9, 16

if __name__ == '__main__':
    start = time.time()
    main()
    print "Multiprocessed: %s seconds" % (time.time()-start)             

    start = time.time()
    for i in range(1,5):
        print i
    print "Normal: %s seconds" % (time.time()-start)

I would imagine that there is some overhead to using `multiprocessing` because it spawns new Python instances to overcome the GIL, which doesn't really happen instantly. If your tasks took a few seconds each, you'd notice a speed boost. — Blender, Aug 22 '12 at 06:35

score 2 · Accepted Answer · answered Aug 22 '12 at 06:55

2

@Blender basically already answered your question, but as a comment. There is some overhead associated with the multiprocessing machinery, so if you incur the overhead without doing any significant work, it will be slower.

Try actually doing some work that parallelizes well. For example, write Python code to open a file, scan it using a regular expression, and pull out matching lines; then make a list of ten big files and time how long it takes to do all ten with multiprocessing vs. plain Python. Or write code to compute an expensive function and try that.

I have used multiprocessing.Pool() just to run a bunch of instances of an external program. I used subprocess to run an audio encoder, and it ran four instances of the encoder at once for a noticeable speedup.

answered Aug 22 '12 at 06:55

steveha

74,789
21
92
117

For just running a `subprocess`, there's no reason to use `multiprocessing` instead of just threads. [`multiprocessing.pool.ThreadPool`](http://stackoverflow.com/a/3386632/344821), though undocumented, has the same API as `multiprocessing.Pool`. – Danica Aug 22 '12 at 07:08
Also, it's worth noting that as a rule of thumb you usually don't want to run any more computationally-heavy processes than you have CPU cores (because context switches slow stuff down), and depending on your task the way you split the work up can have significant effects on efficiency. Parallelism is complicated. – Danica Aug 22 '12 at 07:09
My audio files would not have encoded measurably faster had I used `multiprocessing.pool.ThreadPool`... but thank you for telling me about it; I'm glad to learn about it, and I hope that it will be documented soon. `multiprocessing.Pool()` automatically gives you the same number of worker processes as you have cores in your system. My four worker processes were running on a four-core processor, so each encoder had a core to itself and I had nearly a 4x speedup. – steveha Aug 22 '12 at 07:19
Obviously I don't know the particulars of your code, but `subprocess.call` and friends release the GIL while waiting for the process to complete, so threads would run just as fast as processes with slightly less overhead and hassle (not that that's necessarily a big deal in your case, just pointing it out). – Danica Aug 22 '12 at 14:20
You don't need to know the particulars of the code! All you need to know is that the audio encoder takes tens of seconds to run, and there were a few dozen audio files. So a few microseconds saved by using `.ThreadPool()` instead of `multiprocessing.Pool()` multiplied by a few dozen files cannot possibly add up to a significant savings. In this case, getting all four cores working on the problem, instead of just one, significantly sped up my script. And I don't understand how the `.ThreadPool()` can have any less hassle, since you said it is used identically to `multiprocessing.Pool()`. – steveha Aug 22 '12 at 18:37
Oh, I misunderstood what you meant: I thought you were saying that a thread pool wouldn't be faster than if you didn't do it in parallel at all. And by hassle I meant not having to worry about arguments being pickled/unpickled, which presumably isn't a problem for audio encoding that's probably operating on data on the filesystem but can be very bothersome if you're operating on big numpy arrays or something. – Danica Aug 22 '12 at 19:42

Multiprocessing takes longer?

1 Answers1