Spark with HDFS without YARN

Asked Apr 08 '16 at 07:43

Active Apr 08 '16 at 08:17

Viewed 617 times

I am trying to configure HDFS for spark. Simple running spark-sub mit with --master spark://IP:7077 --deploy-mode cluster.... ends up with

16/04/08 10:16:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Then it end work. I downloaded and lanuched Hadoop cluster, for testing purposes only one machine. I also set envirtoment variables, althouth I think that I forget about some of them. In the fact I set:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:/usr/local/hadoop/bin:/usr/local/spark/bin  
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/

Could you try to help me ?
I followed it: http://www.ccs.neu.edu/home/cbw/spark.html

edited Apr 08 '16 at 08:17

asked Apr 08 '16 at 07:43

what's the question here ? – eliasah Apr 08 '16 at 08:08
1

I don't know why I get this error. If I set all needed enviroment varibals ? – Apr 08 '16 at 08:10
please update with the error message ! – eliasah Apr 08 '16 at 08:11
Possible duplicate of [Hadoop "Unable to load native-hadoop library for your platform" warning](http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-warning) – eliasah Apr 08 '16 at 08:16
No, it doesnt help – Apr 08 '16 at 08:17
Can you update your question with actual error message ??? – eliasah Apr 08 '16 at 08:18
1

Then first it's a duplicate, a secondly it's just a warning. Your project wont fail from it. – eliasah Apr 08 '16 at 08:23
Well I'm getting this error all the time, and it doesn't stop application from working. – avloss Apr 08 '16 at 14:20

Spark with HDFS without YARN

0 Answers0