0

Environment: Openstack Ubuntu14.04vm 16 virtual CPUs. I wrote a python program which is CPU intensive. It used scipy.optimize tools. I did not use any multiprocess/multithreading. But when running it, the python process generates 17 threads and all CPUs are almost fully used.

Run "top" command and you can see the PID 4152 python process and the all CPUs are used, like this

Run "top -H -p 4152" and you can see the threads, like this

So why is python running on multicores and multithreads, considering I did not do anything about that?

fringsoo
  • 1
  • 1

2 Answers2

2

It happens because scipy.optimize uses BLAS for matrix multiplication, etc. Ubuntu 14.04 uses OpenBLAS by default, and OpenBLAS uses all available cores when possible by default.

I think you can set OPENBLAS_NUM_THREADS=1 environment variable if you want it to be single-threaded.

Mikhail Korobov
  • 21,908
  • 8
  • 73
  • 65
  • Setting OPENBLAS_NUM_THREADS=1 in ~/.bashrc does not work. Still running on multicore. How can I check if I am using OpenBLAS or other libraries? – fringsoo Nov 16 '16 at 17:38
  • Hm, maybe I'm wrong and OpenBLAS is not a default. To check library numpy is using see http://stackoverflow.com/questions/9000164/how-to-check-blas-lapack-linkage-in-numpy-scipy – Mikhail Korobov Nov 16 '16 at 17:51
  • OK, thanks! Problem solved. Seems BLAS/LAPACK is the default. – fringsoo Nov 17 '16 at 07:39
2

scipy uses external libraries (most notably BLAS and LAPACK) for a lot of common operations. Depending on the implementation of these libraries on your system, a lot of operations are optimized for multiprocessing.

Sven Rusch
  • 1,357
  • 11
  • 17