5

I was recently able to build Apache Hadoop 2.5.1 with native 64 bit support. So, I got rid of the annoying Native Libraries Warning.

I'm trying to configure Apache Spark. When I start spark-shell, the same warning appears: 

14/09/14 18:48:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Some tips:

I had to download a pre-built 2.4 version of Spark because there is still no profile for Hadoop 2.5 with Maven.

The following exports were added to spark-env.sh:

export HADOOP_CONF_DIR=/opt/hadoop-2.5.1/etc/hadoop

export SPARK_LIBRARY_PATH=/opt/hadoop-2.5.1/lib/native

Didn't work with spark-shell and spark-submit. My Hadoop local installation is configured as pseudo-distributed (ResourceManager + YARN support).

Manjunath Ballur
  • 6,287
  • 3
  • 37
  • 48
hsparks
  • 73
  • 1
  • 5

1 Answers1

2

You should add HADOOP_HOME/lib/native to the LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native
zero323
  • 322,348
  • 103
  • 959
  • 935