Why python multiprocess pool is slower than one process only?

Question

time_1 = time.time()
num = 1000000
for i in range(num):
    print i
time_2 = time.time()
print time_2 - time_1

13.1949999332

def time_test_pool(num):
    print num

if __name__ == "__main__":
    time_1 = time.time()
    num = 1000000
    pool = ThreadPool(8)
    pool.map(time_test_pool, range(num))
    pool.close()
    pool.join()
    time_2 = time.time()
    print time_2 - time_1

15.8250000477

Did I misunderstand the usage of pool? Why the pool is so slow?

Print may be I/o bound printing to the screen. When you run the script, redirect stdout to /dev/null and write the final time to stderr. — tdelaney, Nov 28 '15 at 22:13

score 1 · Accepted Answer · edited May 23 '17 at 12:22

Python threads are not actually running in parallel but time-sliced. The reason is because the python interpreter is not thread-safe.

Said that, python threads are handy when you need to do lots of IO-bound stuff, but it will simply add overheads when trying to perform CPU-bound tasks (like yours).

The solution is typically using python.multiprocess (see python multi-threading slower than serial?), however in your case I am not sure it is going to improve things since the amount of work you do per-thread is very little and the only thing you are paying is the context switch.

Try to assign more iterations per thread, or consider using C++ which supports real multi-threading.

Why python multiprocess pool is slower than one process only?

1 Answers1