How to limit number of cores with threading

Question

The code posted below initiates a single Thread that fires up 4 cores on my macbookpro. Is there a way to limit to how many cores the thread should be using?

    import threading

    import logging
    logging.basicConfig(level=logging.DEBUG, format='(%(threadName)s) %(message)s',)

    def func():
        logging.debug('starting')
        m=0
        for i in range(500000000):
            m+=i
        logging.debug('exiting') 

    thread = threading.Thread(target=func)
    thread.start()

    logging.debug('completed')

Here is the log:

Loaded sitecustomize.py
(Thread-1) starting
(MainThread) completed
(Thread-1) exiting
[Finished in 128.0s]

score 5 · Accepted Answer · answered Dec 30 '16 at 20:33

There is multithreading and multiprocessing.

You can bind a process to a cpu-core, but you can't bind a thread to a core.

The main difference between processes and threads are creation time, threads spawn faster and they run in the same memory space, while processes have separate memory.

import multiprocessing as mp
import psutil


def spawn():
    procs = list()
    n_cpus = psutil.cpu_count()
    for cpu in xrange(n_cpus):
        affinity = [cpu]
        d['affinity'] = affinity
        p = mp.Process(target=run_child, kwargs=d)
        p.start()
        procs.append(p)
    for p in procs:
        p.join()
        print('joined')

def run_child(affinity):
    proc = psutil.Process()  # get self pid
    print('PID: {pid}'.format(pid=proc.pid))
    aff = proc.cpu_affinity()
    print('Affinity before: {aff}'.format(aff=aff))
    proc.cpu_affinity(affinity)
    aff = proc.cpu_affinity()
    print('Affinity after: {aff}'.format(aff=aff))


if __init__ == '__main__':
    spawn()

[The docs](https://pythonhosted.org/psutil/#psutil.Process.cpu_affinity) don't say it's available on OSX. — Sean McSomething, Dec 30 '16 at 21:43
Only "Linux, Windows, FreeBSD" are listed as having support right now. — Sean McSomething, Dec 30 '16 at 22:08
Yes, I thought a macbookpro can run all of them OSes even a Virtual Machine. I didn't read OSX anywhere in the question, did you? — Michael D., Dec 30 '16 at 22:24

score 3 · Answer 2 · answered Jan 10 '17 at 02:21

You can set process CPU affinity in Python with psutil:

>>> import psutil
>>> psutil.cpu_count()
4
>>> p = psutil.Process()
>>> p.cpu_affinity()  # get
[0, 1, 2, 3]
>>> p.cpu_affinity([0])  # set; from now on, process will run on CPU #0 only
>>> p.cpu_affinity()
[0]
>>>

score 2 · Answer 3 · edited May 23 '17 at 12:17

2

since you are using multithreading it is actually is a single process. so the problem is how to make that process pin to a cpu, i.e. How to set processor affinity on OS X? a lot of things are explained in this question, althou the symptom is not same as yours.

edited May 23 '17 at 12:17

Community

1
1

answered Dec 30 '16 at 19:19

Dyno Fu

8,753
4
39
64

score 2 · Answer 4 · edited Mar 20 '17 at 10:04

What you're asking about is called "Processor Affinity". Normally, the operating system's scheduler takes care of this and it's something you shouldn't worry about. Yes, your thread is bouncing between 4 cores but it's only really using about 25% of each one.

Here's a SuperUser question about setting processor afinity in OSX. It doesn't look like there's anything to fiddle with it in the Python standard library since there's very few practical reasons to do so.

No, wanting your CPU usage charts to look prettier is not a valid reason to set CPU affinities.

How to limit number of cores with threading

4 Answers4

Linked