0

I have spark 1.6.1 and a windows 8 machine.

When I run the command c:\spark-1.6.1-bin-hadoop2.6\spark-1.6.1-bin-hadoop2.6>bin\spark-shell there is a lot of screen output and then it gives me the below message. How could i get rid of the error?

Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
16/03/15 08:48:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/03/15 08:48:27 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NullPointerException
        at java.lang.ProcessBuilder.start(Unknown Source)
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
        at org.apache.hadoop.util.Shell.run(Shell.java:455)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:808)
        at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
        at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)
        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:582)
        at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:557)
        at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:599)
        at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
        ... 62 more
<console>:16: error: not found: value sqlContext
         import sqlContext.implicits._
                ^
<console>:16: error: not found: value sqlContext
         import sqlContext.sql
                ^
user2543622
  • 5,760
  • 25
  • 91
  • 159
  • 1
    Possible duplicate of [Failed to locate the winutils binary in the hadoop binary path](http://stackoverflow.com/questions/19620642/failed-to-locate-the-winutils-binary-in-the-hadoop-binary-path) – Yuval Itzchakov Mar 15 '16 at 15:50
  • But i dont have a hadoop cluster. I have a single windows machine – user2543622 Mar 15 '16 at 15:51
  • Doesn't matter, it's not resolving the path to your winutils.exe. Solution should be the same. – Yuval Itzchakov Mar 15 '16 at 15:52
  • You need to set up an environment variable called HADOOP_HOME where the value points to the path containing winutils – Yuval Itzchakov Mar 15 '16 at 16:43
  • I have downloaded the file `hadoop-common-2.2.0-bin-master.zip`. Do i have to unzip it? I have set the enviroment variable hadooop_home too – user2543622 Mar 15 '16 at 17:03
  • i unzipped the file and still getting the same error :( – user2543622 Mar 15 '16 at 17:05
  • @YuvalItzchakov can you unvote the question? it is not a repeat. The link that you showed is a very old and the solution doesnt work. Otherwise please reply my earlier comments – user2543622 Mar 15 '16 at 22:06
  • The link you posted in your answer says to do the same thing as the duplicate answer I pointed you to tells you. I'm not sure how they're different? – Yuval Itzchakov Mar 16 '16 at 07:34
  • it has clear instructions that i was able to follow. Further it points to a newer version of winutils.exe – user2543622 Mar 16 '16 at 13:51

1 Answers1

0

The link from Microsoft and the stackoverflow link that is given at the bottom of the page solves the problem

user2543622
  • 5,760
  • 25
  • 91
  • 159