1

I am new to Hadoop and learnt that with 2.x version, I can try Hadoop on my local Windows 7 64-bit machine.
I installed hadoop 2.6.0 and installed cygwin.
I could execute bin/hadoop version but I get the below error while executing the jar command:
Note: I have also placed the winutils.jar in the bin, from hadoop-common-2.2.0.jar.
Please help. I am not able to get rid of this error. I have also entered the input and output parameters, it still fails.

$ bin/hadoop jar /Hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount
15/02/03 12:40:45 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)
at org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows
(GenericOptionsParser.java:438)
at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions
(GenericOptionsParser.java:484)
at org.apache.hadoop.util.GenericOptionsParser.<init>
(GenericOptionsParser.java:170)
at org.apache.hadoop.util.GenericOptionsParser.<init>   
(GenericOptionsParser.java:153)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke
(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Usage: wordcount <in> [<in>...] <out>

I could run the below command as well:

$ bin/hadoop jar /Hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar
Dharmesh Porwal
  • 1,406
  • 2
  • 12
  • 21
Swati Gupta
  • 73
  • 2
  • 3
  • 8

2 Answers2

0

It used to be an issue earlier. However if you are able to run the program through jar, there could be something else at fault. If the same thing works for you using a Java code, you can edit the jar to remove the code where a new exception is being raised.

To be doubly sure, check if the bin directory contains winutils.exe and hadoop.dll.

If they are not present, chances are that someone else must have faced a similar issue and would have kept the files. These files are created when Hadoop is built from source code on the OS.

Harman
  • 751
  • 1
  • 9
  • 31
  • Hi, I had manually copied winutils.exe .hadoop.dll was not present, I copied it but still the same error. Is it that I need to set the HADOOP_HOME somehwere since it looks for null\bin\winutils.exe at a null location? – Swati Gupta Feb 03 '15 at 06:13
  • That shouldn't be the case, since jar is running ok. Try setting HADOOP_HOME to your HADOOP installation directory. – Harman Feb 03 '15 at 06:20
  • Also, from where did you copy the winutils ? – Harman Feb 03 '15 at 06:20
  • I did try setting the HADOOP_HOME on the cygwin client as well as the hadoop-env.sh, but did not help. I copied the winutils from one of the zip I found online named hadoop-common-2.2.0-bin-master.zip. – Swati Gupta Feb 03 '15 at 06:31
  • Surprisingly, I tried running the sudoku job, and it did not give any error. $ bin/hadoop jar /Hadoop/hadoop-2.6.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar sudoku Include a puzzle on the command line. So, its something to do with GenericOptionsParser, I assume – Swati Gupta Feb 03 '15 at 06:34
  • Can you try executing a simple short java code which would read a file from HDFS and write it to some other location in HDFS. Also, check if HADOOP_HOME env variable has taken effect – Harman Feb 03 '15 at 07:07
  • Are there any existing programs which I need to write? Since, I am very new to Hadoop and not much aware of HDFS.. which is why I wanted to get the infrastructure up and running.. Ideally, this should work on Windows right?Just to confirm if I am going in the right direction? – Swati Gupta Feb 03 '15 at 08:14
  • In an ideal world, every piece of code would run everywhere flawlessly :P The thing is, I haven't worked on windows but using cygwin just gives you a virtual environment where you can run Linux commands and internally they can be binded to Windows calls. However, following this architecture needs some special executable files, which is the exception in your case – Harman Feb 03 '15 at 08:32
  • You can write a word count which is considered to be "Hello World" for map reduce. You'll easily find the program over the web. – Harman Feb 03 '15 at 08:35
  • Thats right.. And I cant find much information on this.. Would you recommend to use Hadoop on Unix or is there any other forum I can confirm this behavior? I am following Hadoop In Action.. – Swati Gupta Feb 03 '15 at 08:36
  • I am not sure of a forum which could confirm this or help you out with cygwin, but yes I will surely recommend moving to Linux (or Unix if you prefer) for Hadoop – Harman Feb 03 '15 at 08:43
  • Also, I would suggest you to quickly gain reputation so as to be able to chat here, and avoid a long comment list. – Harman Feb 03 '15 at 08:44
0

It seems like that you have installed hadoop 2.6.0 and older version of hadoop winutils. You must install hadoop winutils of your current hadoop version. Try to download winutils from this github repo https://github.com/steveloughran/winutils/tree/master/hadoop-2.6.0/bin Finally replace your bin directory with the winutils bin directory!

Ayesha Shabbir
  • 1
  • 1