11

Both os and multiprocessing modules define a cpu_count function.

os.cpu_count is documented as follows:

Return the number of CPUs in the system. Returns None if undetermined.

and multiprocessing.cpu_count's documentation says:

Return the number of CPUs in the system. May raise NotImplementedError. See also os.cpu_count()

On my machine, they both return the same result:

>>> import os
>>> import multiprocessing as mp
>>> os.cpu_count()
8
>>> mp.cpu_count()
8

I would have thought that multiprocessing.cpu_count would be a mere reference to os.cpu_count, but it is not:

>>> os.cpu_count is mp.cpu_count
False

So what is the difference between them? Am I guaranteed that they'll always return the same result? Moreover, if I want to specify a number of processes to create for multiprocessing.Pool, should I use os or multiprocessing's function?

Right leg
  • 16,080
  • 7
  • 48
  • 81

1 Answers1

16

The answer lies in multiprocessing.context, which defines BaseContext.cpu_count as follows:

# cpython/Lib/multiprocessing/context.py

class BaseContext(object):
    def cpu_count(self):
        '''Returns the number of CPUs in the system'''
        num = os.cpu_count()
        if num is None:
            raise NotImplementedError('cannot determine number of cpus')
        else:
            return num

Then, this cpu_count method is exposed by multiprocessing:

# cpython/Lib/multiprocessing/__init__.py

__all__ = [x for x in dir(context._default_context) if not x.startswith('_')]
globals().update((name, getattr(context._default_context, name)) for name in __all__)

So in the end, multiprocessing.cpu_count is nothing but a wrapper around os.cpu_count.

Right leg
  • 16,080
  • 7
  • 48
  • 81
  • 1
    I would like to add that if you are dealing with typing, it's useful to use `mp.cpu_count()` because you won't have to deal with the NoneType in testing – jwillis0720 Mar 29 '22 at 04:22