25

I am using Enthought's Canopy/EPD version of python which ships with numpy linked against MKL. I am currently running a set of simulations in parallel (using PP) on both my work computer (Windows 7, Quad Core i5 @ 3.33 Ghz, 4 GB ram) and my home workstation (Mac Pro 3.1, Ubuntu 12.04, 2x Quad Core Xeon @ 2.8 Ghz, 6 GB ram).

But when I benchmark my simulations, they run much quicker on the work computer (35 seconds per iteration vs. 60 on the mac pro). The problem is being perfectly balanced between cores (embarrassingly parallel problem), so I suspect there is an issue with the MKL library on the linux workstation at home. Is there a way to verify that the MKL library is actually being used in python. I've read threads that say you can check to see if python is linked to it, but it doesn't ensure that it was built correctly and is actually being used.

user1554752
  • 707
  • 2
  • 10
  • 24
  • 1
    Is `scipy.show_config()` not what you're looking for? Not sure if this counts as "ensuring" that it's actually being used. – askewchan Mar 25 '14 at 20:36
  • when I type that, all of the 'library_dirs' and 'include_dirs' say: "/home/vagrant/src/master-env/...". Is that correct (my user account is not 'vagrant' and when i check for that folder, it is not there). Is this field important, or not used? The libraries correctly report "mkl_lapack95_lp64" etc. – user1554752 Mar 25 '14 at 20:47

1 Answers1

31
>>> numpy.show_config()

You will see output something like this, showing that MKL is indeed linked.

lapack_opt_info:
    libraries = ['mkl_lapack95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
    library_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../include']
blas_opt_info:
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
    library_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../include']
openblas_info:
NOT AVAILABLE
lapack_mkl_info:
    libraries = ['mkl_lapack95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
    library_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../include']
blas_mkl_info:
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
    library_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../include']
mkl_info:
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']
    library_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../lib']
    define_macros = [('SCIPY_MKL_H', None)]
    include_dirs = ['/Users/vagrant/src/master-env/Resources/Python.app/Contents/MacOS/../../../../include']
Jonathan March
  • 5,800
  • 2
  • 14
  • 16
  • Regarding your underlying concern, have you taken into account physical cores vs logical cores? You could try tweaking environment variable MKL_NUM_THREADS to avoid possible thrashing. – Jonathan March Mar 25 '14 at 22:24
  • I'll look at that again, but the operations I'm doing aren't multi-threaded (a lot of add(), multiply(), etc. but no dot()). Also, I'm not sure why this would be a problem on the XEON but not on the i5 – user1554752 Mar 26 '14 at 00:15
  • these paths are hardcoded lists you can find in __config__.py and not necessarily the what numpy uses. – Fnord Aug 13 '19 at 17:20