4

I am using the scipy minimize function. The function that it's calling was compiled with Cython and has an underlying C++ implementation that I wrote, but that shouldn't really matter. For some reason when I run my program, it creates as many threads as it can to fill all my cpus. For example if I run top I see that 800% of a cpu is being used or on htop I can see that 8 individual processors are being used, when I only created the program to be run on one. I didn't think that scipy even had parallel processing functionality and I can't find any documentation related to this. What could possible be going on and is there any way to control it?

baconeer
  • 211
  • 1
  • 2
  • 7

1 Answers1

6

If some BLAS-implementation (with threading-support) is available (default on Ubuntu for example), some expressions like np.dot() (only the dense case as far as i know) will automatically be run in parallel (reference). Another possible example is sparse-matrix factorization with SuperLU.

Of course different minimizers will behave different.

Newton-type methods (core: solve a system of sparse linear-equations) are probably based on SuperLU (if the code is not one of the common old Fortran/C ones, where the whole code is self-contained). CG-type methods are heavily based on matrix-vector products (np.dot; so the dense-case will be parallel).

For some control over this, start with this SO question.

Community
  • 1
  • 1
sascha
  • 32,238
  • 6
  • 68
  • 110
  • 1
    This is precisely what was happening. I had no idea that some implementations of BLAS automatically run in parellel. Setting `MKL_NUM_THREADS` to 1 solved the problem. Thanks. – baconeer Feb 24 '17 at 19:35
  • @baconeer A few years ago, the shipped BLAS of ubuntu was single-thread (or maybe 2-thread) only. This changed then. As the linked SO-answer shows, it's not always easy to analyze all the stuff linked within numpy/scipy. – sascha Feb 24 '17 at 20:03