2

How can I change the MKL (Math Kernel Library) version used by NumPy and Miniconda?

Intel's MKL doesn't perform well because on AMD processors, as MKL deliberately chooses the slowest path on non-intel CPUs, "crippling" numerical processes on AMD processors. This is very problematic for scientific work on AMD CPUs where numerical calculations using NumPy are greatly affected by this "cripple AMD" functionality.

The Python I am using (Python 3.9.5) is distributed by Miniconda, and NumPy was installed using conda install numpy. This installed NumPy with MKL version 2021.0.3.

Previously a workaround for the "cripple AMD" function was to set an environment variable MKL_DEBUG_CPY_TYPE=5. But starting MKL 2020, this variable was removed (intel, why??), so it was no longer able to get this workaround working.

So the solution now is to downgrade NumPy's (and conda's) MKL version to 2019. How can this be achieved? How can I change the MKL version used by NumPy and Conda, from 2021.0.3 to 2019?

System info:

  • Python: 3.9.5
  • conda: 4.10.3
  • NumPy: 1.20.3
  • MKL (mkl.get_version_string()): 2021.3
  • NumPy MKL (np.__mkl_version__): 2021.0.3

Please let me know because this is a critical issue for scientific computation on AMD CPUs.

Thank you in advance!!!

PS: Before you say "MKL is written by Intel for Intel processors, so its ok to cripple other processors!", please keep in mind that there should be a competitive spirit during competition, such as amazing innovations, not anti-competitive actions like deliberately slowing down performance on competitor's CPUs . If you want to win a race, practice and improve your running technique, don't break your competitor's legs.

Please avoid a debate, and try to answer my question if you can. If you can't, just ignore and leave.

AstroTeen
  • 191
  • 4
  • 12
  • Have you tried `conda install mkl=2019.*`? Also, why not OpenBLAS? – merv Aug 29 '21 at 22:06
  • @merv thanks for your comment. I have not tried it. Will it replace MKL 2021.3 with 2019.x? If it will, I will give it a try. Also, OpenBLAS is way slower than MKL, slower even than the default ,crippling, MKL (https://i.stack.imgur.com/uVJV8.png) So OpenBLAS is not an option, atleast for now. Thanks again? – AstroTeen Aug 30 '21 at 03:24
  • @merv I have installed MKL 2019.4 using `conda`. Now how do I change NumPy's MKL version? – AstroTeen Aug 30 '21 at 06:39
  • Just to be clear, I run a 5900X. But man, you cannot *debate* an issue in your question (which should be purely technical) and then ask not to debate. Intel and Nvidia spend a LOT of man-hours and money developing these libraries. AMD does NOT. ROCm is a mess. So I think it's more than fair for Intel to reserve its libraries for its own processors. Now, coming to the technical side, I use OpenBLAS (you don't it? You should have bought Intel). – MadHatter Jan 26 '22 at 21:34

2 Answers2

5

I would make a new environment, and probably source from the Anaconda channel. The following works for me:

Bash

## create environment
conda create -n foo -c anaconda python numpy mkl=2019.* blas=*=*mkl

## activate and launch python
conda activate foo
python

Python

import mkl
import numpy as np

mkl.get_version()
## 'Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications'

np.__mkl_version__
## '2019.4'

np.show_config()
## blas_mkl_info:
##     libraries = ['mkl_rt', 'pthread']
##     library_dirs = ['/Users/mfansler/miniconda3/envs/foo/lib']
##     define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
##     include_dirs = ['/Users/mfansler/miniconda3/envs/foo/include']
## blas_opt_info:
##     libraries = ['mkl_rt', 'pthread']
##     library_dirs = ['/Users/mfansler/miniconda3/envs/foo/lib']
##     define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
##     include_dirs = ['/Users/mfansler/miniconda3/envs/foo/include']
## lapack_mkl_info:
##     libraries = ['mkl_rt', 'pthread']
##     library_dirs = ['/Users/mfansler/miniconda3/envs/foo/lib']
##     define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
##     include_dirs = ['/Users/mfansler/miniconda3/envs/foo/include']
## lapack_opt_info:
##     libraries = ['mkl_rt', 'pthread']
##     library_dirs = ['/Users/mfansler/miniconda3/envs/foo/lib']
##     define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
##     include_dirs = ['/Users/mfansler/miniconda3/envs/foo/include']

Notes

I'm leaving this here as an "as-is" answer, since there seems to be some complications that are beyond me. Namely, Anaconda and Conda Forge appear to have different integration strategies when it comes to NumPy + MKL. Anaconda builds NumPy with direct integration (including the np.__mkl_version__ extension); Conda Forge appears to generically build NumPy with BLAS/LAPACK, and in turn builds libblas, liblapack variants based on MKL implementations. Not sure what differences these strategies might make.

Anaconda channel only has Python 3.8 at this point - but that fits with the intended MKL 2019.* anyway. Python 3.9 was a late 2020 release.

The blas=*=*mkl is crucial: that is what constrains to using an MKL build for NumPy.

This was on osx-64 platform - hopefully, the differences are not substantial.

merv
  • 67,214
  • 13
  • 180
  • 245
2

According to conda-forge documentation:

You can switch your BLAS implementation by doing,

conda install "libblas=*=*mkl"
conda install "libblas=*=*openblas"
conda install "libblas=*=*blis"
conda install "libblas=*=*accelerate"
conda install "libblas=*=*netlib"

This would change the BLAS implementation without changing the conda packages depending on BLAS.

The following legacy commands are also supported as well.

conda install "blas=*=mkl"
conda install "blas=*=openblas"
conda install "blas=*=blis"
conda install "blas=*=accelerate"
conda install "blas=*=netlib"
Maurício Collaça
  • 401
  • 3
  • 7