Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so

Question

I am running a python script and I get this error:

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

Both files are present in the anaconda2/lib directory. How can I fix this error? Thanks.

Answer 1

If you use conda, try with these two commands:

conda install nomkl numpy scipy scikit-learn numexpr
conda remove mkl mkl-service

It should fix your problem.

Answer 2

I ran into this problem after installing anaconda3 (vesion 4.2.0). The fix for me was simple, and I was able to keep using mkl. Just update to the latest numpy version.

conda update numpy

Answer 3

Just wanted to note that Anaconda 4.0.0, shipped with mkl enabled by default, has this issue. The problem is indeed with Anaconda, as it can be reproduced with the simple python test below.

The actual issue is that Anaconda linked with mkl, but not with libmkl_core.so, thus it has a missing symbol, and can be seen by running:

$ LD_DEBUG=symbols python -c 'import sklearn.linear_model.tests.test_randomized_l1' 2>&1 | grep -i error
      2200:     /opt/anaconda/lib/python2.7/site-packages/scipy/special/../../../../libmkl_avx.so: error: symbol lookup error: undefined symbol: mkl_dft_fft_fix_twiddle_table_32f (fatal)

I didn't want to uninstall mkl, as I'd like to have the performance boost, so I found a workaround which worked for me - preload libmkl_core.so before execution.

$ python -c 'import sklearn.linear_model.tests.test_randomized_l1'
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.
$
$ LD_PRELOAD=/opt/anaconda/lib/libmkl_core.so python -c 'import sklearn.linear_model.tests.test_randomized_l1'
$

Answer 4

I had this same issue using scikit-learn 0.19 and numpy 1.13.3 when running MLPRegressor (and also with a package called pyearth running an algorithm called MARS). I believe the root of the problem was that our python is part of an Anaconda install, but scikit-learn and numpy were installed via pip, and their expectations for mkl must not agree.

Unfortunately my framework is managed by some dedicated company admins, not by me, so I haven't gotten my guy to try recompiling numpy yet. But I was able to find a workaround based on this thread: Adding export LD_PRELOAD=/path/to/anaconda/lib/libmkl_def.so:/path/to/anaconda/lib/libmkl_avx.so:/path/to/anaconda/lib/libmkl_core.so:/path/to/anaconda/lib/libmkl_intel_lp64.so:/path/to/anaconda/lib/libmkl_intel_thread.so:/path/to/anaconda/lib/libiomp5.so to my ~/.bashrc causes the problem to disappear. It's super hacky, and I'd be lying if I said I knew exactly what it's doing (but this is helpful), so I'm hoping a recompile of numpy is a cleaner fix. But at least it works.

Note it is better performance-wise to have the versions of these packages that use mkl. Installing the nomkl versions is a workaround but not a true solution.

Answer 5

Got the same problem and solved by using the command:

conda install nomkl

I got the solution from this discussion https://github.com/BVLC/caffe/issues/3884

Answer 6

It is also possible that you are running Python in a folder which does not exist.

Solution is simply go to another folder.

See this question: sh: 0: getcwd() failed: No such file or directory on cited drive

Answer 7

try export LD_DEBUG=libs and execute your program again.

It will tell you which .so is causing the problem.

Then you can do something, e.g. upgrading the package.

For me, I find upgrade numexpr useful as it prints the following message just before the error:

calling init: xxxxx/python3.6/site-packages/numexpr/interpreter.cpython-36m-x86_64-linux-gnu.so

Answer 8

Wanted to add on Valilutzik and Zaikun Xu's thread. I can't add comments since I don't have enough points yet.
conda install nomkl numpy scipy scikit-learn numexpr worked for me without having to remove mkl and mkl-service.

Adding an answer to Lee's question if nomkl will be slower: mkl is a Intel math kernel library and is hand-optimized for intel cpus. nomkl uses OpenBlas according to this: https://docs.continuum.io/mkl-optimizations/ It seems that mkl is quite faster on many matrix operations for intel cpus (see https://software.intel.com/en-us/articles/performance-comparison-of-openblas-and-intel-math-kernel-library-in-r)
I've seen someone saying nomkl is faster for AMD cpus (maybe because mkl doesn't work correctly in AMD?)

Answer 9

All the solutions provided above did not work for me, but I found a good compromise:

for someone who has the same error and want's to keep Anaconda on his computer, and steel use mkl for (numpy and scipy) good process performance, the solution I propose:

Edit your .bashrc file.

Look for something like export PATH="/home/anaconda2/bin:$PATH" in the file.

Put a # at the beginning to comment it from the script: #export PATH="/home/anaconda2/bin:$PATH"

Open a new terminal and you should be running the base python installation.

It worked for me, hope it will be helpfull.

Answer 10

In case anyone has a similar issue and gets errors that libmkl_p4m.so or libmkl_p4.so cannot be found (this occurred for me when calling certain numpy functions), I tried reinstalling / updating different python modules and reverting / updating to various versions of Anaconda, and neither worked. However I found that completely uninstalling anaconda and re-installing it (to version 4.4.10) solved the issue.

Answer 11

The following worked for me

conda install  -f  numpy

Answer 12

None of the above answers worked for me. In my case, @Nehal J Wani's comment below the OP's question led to the solution: As stated in the linked Intel support document, a single dynamic library can be used to link to the MKL.

After editing my project's Makefile to replace the following libraries -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core with -lmkl_rt and re-building, the error disappeared.

Answer 13

For me downgrading mkl solved it:

conda install mkl=2021.2.0

(Ubuntu 21.04, Python 3.8)

Answer 14

solved by reinstallation: pip install tensorflow