1

I am using word2vec in mllib and I want to improve the speed of linear operations in word2vec by configuring OpenBLAS.

I configured OpenBLAS on the server by manually compiling OpenBLAS with the following command.

git clone https://github.com/xianyi/OpenBlas.git
cd OpenBLAS
make USE_THREAD=0 -j16
make PREFIX=/usr/local install

sudo update-alternatives --install /usr/lib/libblas.so libblas.so /usr/local/lib/libopenblas.so 5
sudo update-alternatives --install /usr/lib/libblas.so.3 libblas.so.3 /usr/local/lib/libopenblas.so 5
sudo update-alternatives --install /usr/lib/liblapack.so liblapack.so /usr/local/lib/libopenblas.so 5
sudo update-alternatives --install /usr/lib/liblapack.so.3 liblapack.so.3 /usr/local/lib/libopenblas.so 5

I compiled and ran the following code to make sure I had OpenBLAS configured correctly:

#include <cblas.h>
#include <stdio.h>

int main() {
  double A[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
  double B[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
  double C[9] = {.5,.5,.5,.5,.5,.5,.5,.5,.5};
  cblas_dgemm(CblasColMajor, CblasNoTrans, CblasTrans,3,3,2,1,A, 3, B, 3,2,C,3);

  for(int i = 0; i < 9; i++)
    printf("%lf ", C[i]);
  printf("\n");
  return 0;
}

Use the following command to compile:

gcc test.c -lblas

The program compiles and runs well and produces the correct results.

I have included the netlib-java library in my spark project build file pom.xml as follows:

<dependency>
    <groupId>com.github.fommil.netlib</groupId>
    <artifactId>all</artifactId>
    <version>1.1.2</version>
    <type>pom</type>
</dependency>

I then ran my spark project locally and still got the following warning:

2022-11-09 02:34:48 WARN  BLAS:61 - Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
2022-11-09 02:34:48 WARN  BLAS:61 - Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
2022-11-09 02:34:48 WARN  APP:66 - com.github.fommil.netlib.F2jBLAS

I would like to know what might be wrong? My server configuration is as follows:

Ubuntu 18.04.1 LTS
spark-2.3.2
Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz

I am using word2vec in mllib and I want to improve the speed of linear operations in word2vec by configuring OpenBLAS.

I configured OpenBLAS on the server by manually compiling OpenBLAS with the following command.

git clone https://github.com/xianyi/OpenBlas.git
cd OpenBLAS
make USE_THREAD=0 -j16
make PREFIX=/usr/local install

sudo update-alternatives --install /usr/lib/libblas.so libblas.so /usr/local/lib/libopenblas.so 5
sudo update-alternatives --install /usr/lib/libblas.so.3 libblas.so.3 /usr/local/lib/libopenblas.so 5
sudo update-alternatives --install /usr/lib/liblapack.so liblapack.so /usr/local/lib/libopenblas.so 5
sudo update-alternatives --install /usr/lib/liblapack.so.3 liblapack.so.3 /usr/local/lib/libopenblas.so 5

I compiled and ran the following code to make sure I had OpenBLAS configured correctly:

#include <cblas.h>
#include <stdio.h>

int main() {
  double A[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
  double B[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
  double C[9] = {.5,.5,.5,.5,.5,.5,.5,.5,.5};
  cblas_dgemm(CblasColMajor, CblasNoTrans, CblasTrans,3,3,2,1,A, 3, B, 3,2,C,3);

  for(int i = 0; i < 9; i++)
    printf("%lf ", C[i]);
  printf("\n");
  return 0;
}

Use the following command to compile:

gcc test.c -lblas

The program compiles and runs well and produces the correct results.

I have included the netlib-java library in my spark project build file pom.xml as follows:

<dependency>
    <groupId>com.github.fommil.netlib</groupId>
    <artifactId>all</artifactId>
    <version>1.1.2</version>
    <type>pom</type>
</dependency>

I then ran my spark project locally and still got the following warning:

2022-11-09 02:34:48 WARN  BLAS:61 - Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
2022-11-09 02:34:48 WARN  BLAS:61 - Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
2022-11-09 02:34:48 WARN  APP:66 - com.github.fommil.netlib.F2jBLAS

I would like to know what might be wrong? My server configuration is as follows:

Ubuntu 18.04.1 LTS
spark-2.3.2
Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz
Rain Chen
  • 11
  • 1

0 Answers0