Speed-up a single task using multi-processing or threading

Question

Is it possible to speed up a single task using multi-processing/threading? My gut feeling is that the answer is 'no'. Here is an example of what I mean by a "single task":

for i in range(max):
    pick = random.choice(['on', 'off', 'both'])

With an argument of 10000000 it takes about 7.9 seconds to complete on my system.

I have a basic grasp of how to use multi-processing and threading for multiple tasks. For example, if I have 10 directories each one containing X number of files that need to be read, I could use create 10 threads.

I suspect that the single task is using only a single process (task manager reports CPU usage is minimal). Is there a way to leverage my other cores in such cases? Or is increasing the CPU/Memory speeds the only way to get faster results?

multiprocessing can help, multiple threads in one process not that much (keyword: "global interpreter lock" aka "GIL") — Michael Butscher, Jun 23 '13 at 15:16
@Michael Butsher: Could you provide an example of how I could break up the code I mentioned using multiprocessing? — Dirty Penguin, Jun 23 '13 at 15:20
You can read http://mikecvet.wordpress.com/2010/07/02/parallel-mapreduce-in-python/ for an example — Michael Butscher, Jun 23 '13 at 15:25
See http://stackoverflow.com/questions/12758952/green-threads-and-thread-in-python — Lucas Holt, Jun 23 '13 at 16:34
if you want all results at once, you could use `result = numpy.random.choice(L, size=max)` or `result = np.take(L, np.random.random_integers(0, len(L)-1, max))`, where `L = ['on', 'off', 'both']`. — jfs, Jun 23 '13 at 18:42

score 8 · Accepted Answer · answered Jun 23 '13 at 16:48

8

Here is a benchmark of your code with and without multiprocessing:

#!/usr/bin/env python

import random
import time

def test1():
    print "for loop with no multiproc: "
    m = 10000000
    t = time.time()
    for i in range(m):
        pick = random.choice(['on', 'off', 'both'])
    print time.time()-t

def test2():
    print "map with no multiproc: "
    m = 10000000
    t = time.time()
    map(lambda x: random.choice(['on', 'off', 'both']), range(m))
    print time.time()-t

def rdc(x):
    return random.choice(['on', 'off', 'both'])

def test3():
    from multiprocessing import Pool

    pool = Pool(processes=4)
    m = 10000000

    print "map with multiproc: "
    t = time.time()

    r = pool.map(rdc, range(m))
    print time.time()-t

if __name__ == "__main__":
    test1()
    test2()
    test3()

And here is the result on my workstation (which is a quadcore):

for loop with no multiproc: 
8.31032013893
map with no multiproc: 
9.48167610168
map with multiproc: 
4.94983720779

Is it possible to speed up a single task using multi-processing/threading? My gut feeling is that the answer is 'no'.

well, afaict, the answer is "damn, yes".

Is there a way to leverage my other cores in such cases? Or is increasing the CPU/Memory speeds the only way to get faster results?

yes, by using multiprocessing. Python can't handle multiple cores by using threading, because of the GIL, but it can rely on your operating system's scheduler to leverage the other cores. Then you can get a real improvement on your tasks.

answered Jun 23 '13 at 16:48

zmo

24,463
4
54
90

1

The answer is "damn, yes" only if you're using a good computer. On my netbook, test3 is slower than test1. – michaelmeyer Jun 23 '13 at 16:55
1

the answer "damn yes" is dependant on the number of cores your computer has. If you make a pool of processes that has the same number of processes that you got cores, then you will gain from it (given you have more than one core). If you don't it will be the same (the scheduler does not do miracles). – zmo Jun 23 '13 at 16:59
1

@zmo: 3:16 might've screamed "Hell, yeah!", but "damn, yes" also works :) Thank you. – Dirty Penguin Jun 23 '13 at 17:10
1

Sweet solution. You don't even have to start and stop the threads yourself. – Nils Werner Jun 23 '13 at 17:13
2

Oddly enough, on my machine, with python3, the results are quite different: `for loop with no multiproc: 25.41977310180664 map with no multiproc: 1.0967254638671875e-05 map with multiproc: 11.853065013885498` – Nils Werner Jun 23 '13 at 17:22
1

@zmo: Isn't it what I'm saying? So if you don't have more than one core it doesn't speed up anything, it is even *slower*. – michaelmeyer Jun 23 '13 at 17:53
1

@doukremt indeed, because the `fork()` and the process pool management is quite costy... @NilsWerner I did not test with python3, but it indeed looks there's something weird happening... – zmo Jun 23 '13 at 18:00
1

@NilsWerner: In Python 3, map() behaves like itertools.imap() from Python 2, it returns an iterator. Apply `list()` to consume the result, to get Python 2 behavior. – jfs Jun 23 '13 at 18:05

Speed-up a single task using multi-processing or threading

1 Answers1

Linked