Numpy suddenly uses all CPUs

Question

Until recently when I used numpy methods like np.dot(A,B), only a single core was used. However, since today suddently all 8 cores of my linux machine are being used, which is a problem.

A minimal working example:

import numpy as np
N = 100

a = np.random.rand(N,N)
b = np.random.rand(N,N)

for i in range(100000):
    a = np.dot(a,b)

On my other laptop it works all fine on a single core. Could this be due to some new libraries?

This morning I updated matplotlib and cairocffi via pip, but that's all.

Any ideas how to go back to a single core?

Edit:

When I run

np.__config__.show()

I get the following output

openblas_info:
    libraries = ['openblas', 'openblas']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    library_dirs = ['/usr/local/lib']
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    library_dirs = ['/usr/local/lib']
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    library_dirs = ['/usr/local/lib']
blas_mkl_info:
  NOT AVAILABLE
blas_opt_info:
    libraries = ['openblas', 'openblas']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
    library_dirs = ['/usr/local/lib']

score 6 · Accepted Answer · answered Jul 29 '16 at 12:58

6

This could be because numpy is linking against multithreaded openBLAS libraries. Try setting the global environment variable to set threading affinity as:

export OPENBLAS_MAIN_FREE=1

# Now run your python script.

Another workaround could to use ATLAS instead of OpenBLAS. Please see this post for more information (https://shahhj.wordpress.com/2013/10/27/numpy-and-blas-no-problemo/). This post proposes some other workarounds as well which might be worth trying.

answered Jul 29 '16 at 12:58

Pankaj Daga

741
5
16

Thanks. I ran the export command in my linux shell and the used python test.py to call the code. However, it did not change anything. I updated my answer with np.__config__.show(), if this is of any help. – physicsGuy Jul 29 '16 at 13:35
Try setting the affinity using the OS module. Something like `os.system(“taskset -p 0x1 %s” %os.getpid())` and then run your code (for reference this is in the post I linked to). – Pankaj Daga Jul 29 '16 at 13:37
If multiprocessing is indeed the problem, than you might want to also try using ATLAS rather than openBLAS. This should be easy to do. Please refer to the post for details on how to make numpy use ATLAS rather than openBLAS. – Pankaj Daga Jul 29 '16 at 13:39
I also followed the wordpress post. Using os.system("taskset -p 0x1 %s" %os.getpid()) gives the same output as in the post, however it does not change the behaviour. – physicsGuy Jul 29 '16 at 15:09
Installing Atlas is not so easy for me, as I have no root permissions. Is there an good guide on how to install Atlas locally and link with numpy? – physicsGuy Jul 29 '16 at 15:10
I guess you would have to compile ATLAS from sources and then you could install it at a custom location. I am not sure if it is possible to do it through the APT system. – Pankaj Daga Jul 29 '16 at 15:17
Also see this post: http://stackoverflow.com/questions/21671040/link-atlas-mkl-to-an-installed-numpy – Pankaj Daga Jul 29 '16 at 15:19
CPU throtteling is activated, so I could not install Atlas. However, I solved my problem by simple re installation of numpy. – physicsGuy Jul 29 '16 at 15:52

score 4 · Answer 2 · answered Jul 26 '18 at 07:03

if you're using OpenBLAS try setting the OPENBLAS_NUM_THREADS environment variable = 1. OpenBLAS uses all CPU cores and is inefficient on small matrices.

export OPENBLAS_NUM_THREADS = 1

You can read more here: https://github.com/numpy/numpy/issues/8120

Numpy suddenly uses all CPUs

2 Answers2