23

On Mac OS X, I compiled Spark from the sources using the following command:

jacek:~/oss/spark
$ SPARK_HADOOP_VERSION=2.4.0 SPARK_YARN=true SPARK_HIVE=true SPARK_GANGLIA_LGPL=true xsbt
...

[info] Set current project to root (in build file:/Users/jacek/oss/spark/)
> ; clean ; assembly
...
[info] Packaging /Users/jacek/oss/spark/examples/target/scala-2.10/spark-examples-1.0.0-SNAPSHOT-hadoop2.4.0.jar ...
[info] Done packaging.
[info] Done packaging.
[success] Total time: 1964 s, completed May 9, 2014 5:07:45 AM

When I started ./bin/spark-shell I noticed the following WARN message:

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

What might be the issue?

jacek:~/oss/spark
$ ./bin/spark-shell
Spark assembly has been built with Hive, including Datanucleus jars on classpath
14/05/09 21:11:17 INFO SecurityManager: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/05/09 21:11:17 INFO SecurityManager: Changing view acls to: jacek
14/05/09 21:11:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jacek)
14/05/09 21:11:17 INFO HttpServer: Starting HTTP Server
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.0.0-SNAPSHOT
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0)
Type in expressions to have them evaluated.
Type :help for more information.
...
14/05/09 21:11:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
...
Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
  • What leads you to believe that there is such a library for OSX? – bmargulies May 09 '14 at 19:46
  • 3
    Nothing. I'm a mere end user who when noticed the warning have expected an answer to my and your questions. Is there such a library? Could I use it with Spark? Please guide/advise. – Jacek Laskowski May 09 '14 at 19:49
  • But then you're not asking a programming question, I submit. – bmargulies May 09 '14 at 19:50
  • Does this answer your question? [Hadoop "Unable to load native-hadoop library for your platform" warning](https://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-warning) – OneCricketeer Nov 17 '22 at 22:13

2 Answers2

20

Supported Platforms of the Native Libraries Guide documentation in Apache Hadoop reads:

The native hadoop library is supported on *nix platforms only. The library does not to work with Cygwin or the Mac OS X platform.

The native hadoop library is mainly used on the GNU/Linus platform and has been tested on these distributions:

  • RHEL4/Fedora
  • Ubuntu
  • Gentoo

On all the above distributions a 32/64 bit native hadoop library will work with a respective 32/64 bit jvm.

It appears that the WARN message should be disregarded on Mac OS X as the native library doesn't simply exist for the platform.

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
  • 2
    is it bad to have to use "builtin-java classes where applicable"? shall I put effort into installing hadoop native libraries (I am already on Linux) – architectonic Nov 13 '15 at 14:23
6

In my experience, if you cd into the /sparkDir/conf and rename the spark-env.sh.template to spark-env.sh, and then set the JAVA_OPTSand hadoop_DIR, it works.

You will also have to edit this /etc/profile line:

export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/:$LD_LIBRARY_PATH
Aaron Gillion
  • 2,227
  • 3
  • 19
  • 31
Kevin
  • 160
  • 2
  • 3
  • What would HADOOP_HOME be in the case where Hadoop isn't installed? (e.g. installed Spark directly from the Apache site but using it to submit jobs to a YARN-based Hadoop cluster remotely (EMR)) – DavidJ Jun 24 '16 at 15:59
  • This fixes the problem, but why? – Jeremy Hajek Jul 26 '19 at 01:59