How do I access all computer cores for computation in python script?

Question

I have a python script that has to take many permutations of a large dataset, score each permutation, and retain only the highest scoring permutations. The dataset is so large that this script takes almost 3 days to run.

When I check my system resources in windows, only 12% of my CPU is being used and only 4 out of 8 cores are working at all. Even if I put the python.exe process at highest priority, this doesn't change.

My assumption is that dedicating more CPU usage to running the script could make it run faster, but my ultimate goal is to reduce the runtime by at least half. Is there a python module or some code that could help me do this? As an aside, does this sound like a problem that could benefit from a smarter algorithm?

Thank you in advance!

score 3 · Accepted Answer · answered Apr 05 '13 at 14:44

There are a few ways to go about this, but check out the multiprocessing module. This is a standard library module for creating multiple processes, similar to threads but without the limitations of the GIL.

You can also look into the excellent Celery library. This is a distrubuted task queue, and has a lot of great features. Its a pretty easy install, and easy to get started with.

score 1 · Answer 2 · answered Apr 05 '13 at 15:00

I can answer a HOW-TO with a simple code sample. While this is running, run /bin/top and see your processes. Simple to do. Note, I've even included how to clean up afterwards from a keyboard interrupt - without that, your subprocesses will keep running and you'll have to kill them manually.

from multiprocessing import Process
import traceback
import logging 
import time

class AllDoneException(Exception):
    pass

class Dum(object):
    def __init__(self):
        self.numProcesses = 10    
        self.logger = logging.getLogger()
        self.logger.setLevel(logging.INFO)
        self.logger.addHandler(logging.StreamHandler())

    def myRoutineHere(self, processNumber):
        print "I'm in process number %d" % (processNumber)
        time.sleep(10)
        # optional: raise AllDoneException  

    def myRoutine(self):
        plist = []
        try:
            for pnum in range(0, self.numProcesses):
                p = Process(target=self.myRoutineHere, args=(pnum, ))
                p.start()
                plist.append(p)
            while 1:
                isAliveList = [p.is_alive() for p in plist]
                if not True in isAliveList:
                    break
                time.sleep(1)
        except KeyboardInterrupt:
            self.logger.warning("Caught keyboard interrupt, exiting.")
        except AllDoneException:
            self.logger.warning("Caught AllDoneException, Exiting normally.")
        except:
            self.logger.warning("Caught Exception, exiting: %s" % (traceback.format_exc()))
        for p in plist:
            p.terminate()

d = Dum()
d.myRoutine()

score 0 · Answer 3 · answered Apr 05 '13 at 14:46

You should spawn new processes instead of threads to utilize cores in your CPU. My general rule is one process per core. So you split your problem input space into the number of cores available, each process getting part of the problem space.

Multiprocessing is best for this. You could also use Parallel Python.

score 0 · Answer 4 · answered Jun 29 '17 at 21:55

Very late to the party - but in addition to using multiprocessing module as reptilicus said, also make sure to set "affinity".

Some python modules fiddle with it, effectively lowering the number of cores available to Python:

https://stackoverflow.com/a/15641148/4195846

score -1 · Answer 5 · answered Apr 05 '13 at 14:44

Due to Global Interpreter Lock one Python process cannot take advantage of multiple cores. But if you can somehow parallelize your problem (which you should do anyway), then you can use multiprocessing to spawn as many Python processes as you have cores and process that data in each subprocess.

How do I access all computer cores for computation in python script?

5 Answers5