2

I want to use MKL libraries provided with Anaconda with Spark. I installed libgfortran

conda install libgfortran

linked libs

ln -s $PATH_TO_ANACONDA/lib/libmkl_rt.so  $PATH_TO_ANACONDA/lib/liblapack.so.3
ln -s  $PATH_TO_ANACONDA/lib/libmkl_rt.so $ PATH_TO_ANACONDA/lib/libblas.so.3

and placed PATH_TO_ANACONDA/lib on LD_LIBRARY_PATH

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PATH_TO_ANACONDA/lib

but when I run Spark ALS example I see

WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK

so i think it didn't work. Is something clearly wrong with this?

Community
  • 1
  • 1
a9fb024f
  • 21
  • 1

1 Answers1

-1

You can use any library as long as you load it to the nodes. Use --packages with the libraries you would like to use

z-star
  • 680
  • 5
  • 6