Failed to locate the winutils binary in the hadoop binary path

Question

I am getting the following error while starting namenode for latest hadoop-2.2 release. I didn't find winutils exe file in hadoop bin folder. I tried below commands

$ bin/hdfs namenode -format
$ sbin/yarn-daemon.sh start resourcemanager

ERROR [main] util.Shell (Shell.java:getWinUtilsPath(303)) - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
    at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
    at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:863)

See http://stackoverflow.com/questions/18630019/running-apache-hadoop-2-1-0-on-windows — harpun, Oct 27 '13 at 17:23
Trying to get rid of that error message I set -Dhadoop.home.dir to a meaningful value. Now I am getting something like (local path replaced by variable name): Could not locate executable ${HADOOP_HOME}\bin\winutils.exe in the Hadoop binaries. And guess what: Setting the variable did not make that binary appear. Also the priority ERROR is wrong, as the application seems to continue. In my eyes that is a bug in Hadoop 2.2.0. — Hiran, Dec 19 '13 at 14:23
There is a (as per today open) jira for this> https://issues.apache.org/jira/browse/HADOOP-10051 — René Nyffenegger, Feb 21 '14 at 06:18
There is a [HADOOP-11003](https://issues.apache.org/jira/browse/HADOOP-11003). — Remus Rusanu, Aug 30 '14 at 19:52
Please accept answer given by Prasad D. He has provided readymade files to be used; thus avoiding efforts. — Kaushik Lele, Jun 18 '15 at 04:22
It's partially duplicate of my answer. Both download and source are links of my blog as mentioned in the answer. — Abhijit, Aug 23 '16 at 18:37
That's a known Hadoop bug with Windows OS. https://issues.apache.org/jira/browse/SPARK-2356 — torina, Apr 25 '17 at 17:21
https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-tips-and-tricks-running-spark-windows.html — Alexandru Marculescu, Aug 02 '18 at 13:28

score 106 · Answer 1 · edited Sep 05 '19 at 14:30

106

Simple Solution: Download it from here and add to $HADOOP_HOME/bin

(Source)

IMPORTANT UPDATE:

For hadoop-2.6.0 you can download binaries from Titus Barik blog >>.

I have not only needed to point HADOOP_HOME to extracted directory [path], but also provide system property -Djava.library.path=[path]\bin to load native libs (dll).

edited Sep 05 '19 at 14:30

Ganesh Jadhav

2,830
1
20
32

answered Dec 10 '14 at 06:45

Prasad D

1,496
1
14
8

This helps to start the hdfs but when I try to put a text file into hdfs it gives error Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsBy teArray(II[BI[BIILjava/lang/String;JZ)V – Kaushik Lele Jun 20 '15 at 14:47
It's partially duplicate of my answer. Both download and source are links of my blog as mentioned in the answer. – Abhijit Aug 23 '16 at 18:36
1

This appears to work with Hadoop 2.7.3 as well. Or is would there be an issue? 1) download from the page above. tar xfvz which gives a Hadoop 2.6 distribution. Then copy bin\winutlis.exe to the 2.73 \bin directory. Instantly, it works. So far - at least. – RichMeister Oct 12 '16 at 23:22

TrnKh · Answer 2 · 2016-01-31T23:48:15.847

29

If you face this problem when running a self-contained local application with Spark (i.e., after adding spark-assembly-x.x.x-hadoopx.x.x.jar or the Maven dependency to the project), a simpler solution would be to put winutils.exe (download from here) in "C:\winutil\bin". Then you can add winutils.exe to the hadoop home directory by adding the following line to the code:

System.setProperty("hadoop.home.dir", "c:\\\winutil\\\")

Source: Click here

edited Jan 31 '16 at 23:48

answered Oct 29 '15 at 20:53

TrnKh

758
1
7
13

1

For me it was enough to put the winutils stuff in `hadoop/bin/bin/`. So just copy the winutils `bin` folder in hadoop's `bin` folder. (`HADOOP_HOME` is set in the env vars) – holzkohlengrill Jun 03 '19 at 07:40
+1 There is one too many slash in the directory path, so I'm leaving this here for myself: `System.setProperty("hadoop.home.dir", "c:\\winutil\\")` – emragins Mar 01 '20 at 20:49

Abhijit · Answer 3 · 2013-11-06T05:02:24.440

27

If we directly take the binary distribution of Apache Hadoop 2.2.0 release and try to run it on Microsoft Windows, then we'll encounter ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path.

The binary distribution of Apache Hadoop 2.2.0 release does not contain some windows native components (like winutils.exe, hadoop.dll etc). These are required (not optional) to run Hadoop on Windows.

So you need to build windows native binary distribution of hadoop from source codes following "BUILD.txt" file located inside the source distribution of hadoop. You can follow the following posts as well for step by step guide with screen shot

Build, Install, Configure and Run Apache Hadoop 2.2.0 in Microsoft Windows OS

ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path

edited Nov 06 '13 at 05:02

answered Nov 03 '13 at 21:10

Abhijit

746
5
18

3

Hi Abhijit. Have you followed your own directions? Hadoop is not supported on Windows. Actually it is only supported on certain versions of Windows Server. That is very lousy by Microsoft and HortonWorks. – WestCoastProjects Jun 29 '14 at 19:37
2

That was over two years ago and very (very!) true at the time. I had verified directly with Hortonworks engineers just two weeks prior at a conference. – WestCoastProjects Aug 23 '16 at 23:20

score 20 · Answer 4 · edited Feb 18 '20 at 17:11

Winutils.exe is used for running the shell commands for SPARK. When you need to run the Spark without installing Hadoop, you need this file.

Steps are as follows:

Download the winutils.exe from following location for hadoop 2.7.1 https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin [NOTE: If you are using separate hadoop version then please download the winutils from corresponding hadoop version folder on GITHUB from the location as mentioned above.]
Now, create a folder 'winutils' in C:\ drive. Now create a folder 'bin' inside folder 'winutils' and copy the winutils.exe in that folder. So the location of winutils.exe will be C:\winutils\bin\winutils.exe
Now, open environment variable and set HADOOP_HOME=C:\winutils [NOTE: Please do not add \bin in HADOOP_HOME and no need to set HADOOP_HOME in Path]

Your issue must be resolved !!

score 19 · Answer 5 · answered Nov 02 '13 at 09:04

19

The statement java.io.IOException: Could not locate executable null\bin\winutils.exe

explains that the null is received when expanding or replacing an Environment Variable. If you see the Source in Shell.Java in Common Package you will find that HADOOP_HOME variable is not getting set and you are receiving null in place of that and hence the error.

So, HADOOP_HOME needs to be set for this properly or the variable hadoop.home.dir property.

Hope this helps.

Thanks, Kamleshwar.

answered Nov 02 '13 at 09:04

Kamleshwar

321
1
4

1

On which configuration file we need to set hadoop.home.dir property (Ex. hdfs-site.xml, core-site.xml ) ? – Sarde Aug 21 '14 at 04:08
4

System.setProperty("hadoop.home.dir", "C:\\winutil\\"); reference : http://stackoverflow.com/a/33610936/3110474 – Himanshu Bhandari Jan 06 '16 at 07:01
If you're using Windows, make sure you use double backslashes while writing any path as a value. – rishirich Dec 04 '19 at 12:02

score 13 · Answer 6 · answered May 27 '15 at 21:02

I just ran into this issue while working with Eclipse. In my case, I had the correct Hadoop version downloaded (hadoop-2.5.0-cdh5.3.0.tgz), I extracted the contents and placed it directly in my C drive. Then I went to

Eclipse->Debug/Run Configurations -> Environment (tab) -> and added

variable: HADOOP_HOME

Value: C:\hadoop-2.5.0-cdh5.3.0

score 8 · Answer 7 · answered Aug 14 '15 at 06:29

8

You can download winutils.exe here: http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe

Then copy it to your HADOOP_HOME/bin directory.

answered Aug 14 '15 at 06:29

Soumya Kanti

1,429
1
17
28

In case the link didn't work, you can also download it from here: https://github.com/steveloughran/winutils/releases – Mohamad Al Mdfaa Dec 30 '21 at 21:09

score 4 · Answer 8 · answered Sep 27 '18 at 16:33

4

In Pyspark, to run local spark application using Pycharm use below lines

os.environ['HADOOP_HOME'] = "C:\\winutils"
print os.environ['HADOOP_HOME']

answered Sep 27 '18 at 16:33

Narsireddy

368
5
10

score 3 · Answer 9 · answered Oct 30 '14 at 11:51

3

winutils.exe are required for hadoop to perform hadoop related commands. please download hadoop-common-2.2.0 zip file. winutils.exe can be found in bin folder. Extract the zip file and copy it in the local hadoop/bin folder.

answered Oct 30 '14 at 11:51

Mohan Raj

1,104
9
17

score 3 · Answer 10 · answered Jan 07 '16 at 10:45

3

I was facing the same problem. Removing the bin\ from the HADOOP_HOME path solved it for me. The path for HADOOP_HOME variable should look something like.

C:\dev\hadoop2.6\

System restart may be needed. In my case, restarting the IDE was sufficient.

answered Jan 07 '16 at 10:45

Asmat Ali

335
1
11

Thank you @Asmat Ali, restart of IDE was the part i was missing. – alterego Dec 23 '21 at 12:33

score 3 · Answer 11 · answered Dec 29 '21 at 20:37

As most answers here refer to pretty old versions of winutils, I will leave a link to the most comprehensive repository, which supports all versions of Hadoop including the most recent ones:

https://github.com/kontext-tech/winutils (find the directory corresponding to your Hadoop version, or try the most recent one).

If you have admin permissions on you machine.

Put bin directory into C:\winutils
The whole path should be C:\winutils\bin\winutils.exe
Set HADOOP_HOME into C:\winutils

If you don't have admin permissions or want to put the binaries into user space.

Put bin directory into C:\Users\vryabtse\AppData\Local\Programs\winutils or similar user directory.
Set HADOOP_HOME value into path to this directory.

score 2 · Answer 12 · answered Oct 28 '16 at 10:23

Set up HADOOP_HOME variable in windows to resolve the problem.

You can find answer in org/apache/hadoop/hadoop-common/2.2.0/hadoop-common-2.2.0-sources.jar!/org/apache/hadoop/util/Shell.java :

IOException from

  public static final String getQualifiedBinPath(String executable) 
  throws IOException {
    // construct hadoop bin path to the specified executable
    String fullExeName = HADOOP_HOME_DIR + File.separator + "bin" 
      + File.separator + executable;
    File exeFile = new File(fullExeName);
    if (!exeFile.exists()) {
      throw new IOException("Could not locate executable " + fullExeName
        + " in the Hadoop binaries.");
    }
    return exeFile.getCanonicalPath();
  }

HADOOP_HOME_DIR from

// first check the Dflag hadoop.home.dir with JVM scope
String home = System.getProperty("hadoop.home.dir");
// fall back to the system/user-global env variable
if (home == null) {
  home = System.getenv("HADOOP_HOME");
}

score 2 · Answer 13 · answered Mar 21 '18 at 09:34

Download desired version of hadoop folder (Say if you are installing spark on Windows then hadoop version for which your spark is built for) from this link as zip.

Extract the zip to desired directory. You need to have directory of the form hadoop\bin (explicitly create such hadoop\bin directory structure if you want) with bin containing all the files contained in bin folder of the downloaded hadoop. This will contain many files such as hdfs.dll, hadoop.dll etc. in addition to winutil.exe.

Now create environment variable HADOOP_HOME and set it to <path-to-hadoop-folder>\hadoop. Then add ;%HADOOP_HOME%\bin; to PATH environment variable.

Open a "new command prompt" and try rerunning your command.

this worked for me, and the alternative with HADOOP_HOME pointing to the bin directory (rather than it's parent) did not. — philwalk, Jul 24 '18 at 16:54

score 2 · Answer 14 · answered Jun 22 '18 at 11:57

2

Download [winutils.exe]
From URL :
https://github.com/steveloughran/winutils/hadoop-version/bin
Past it under HADOOP_HOME/bin
Note : You should Set environmental variables:
User variable:
Variable: HADOOP_HOME
Value: Hadoop or spark dir

answered Jun 22 '18 at 11:57

Mostafa M.Shawky

41
2

score 1 · Answer 15 · answered Mar 03 '17 at 11:05

I used "hbase-1.3.0" and "hadoop-2.7.3" versions. Setting HADOOP_HOME environment variable and copying 'winutils.exe' file under HADOOP_HOME/bin folder solves the problem on a windows os. Attention to set HADOOP_HOME environment to the installation folder of hadoop(/bin folder is not necessary for these versions). Additionally I preferred using cross platform tool cygwin to settle linux os functionality (as possible as it can) because Hbase team recommend linux/unix env.

score 1 · Answer 16 · answered Sep 15 '17 at 06:45

I was getting the same issue in windows. I fixed it by

Downloading hadoop-common-2.2.0-bin-master from link.
Create a user variable HADOOP_HOME in Environment variable and assign the path of hadoop-common bin directory as a value.
You can verify it by running hadoop in cmd.
Restart the IDE and Run it.

score 0 · Answer 17 · answered Mar 23 '22 at 20:23

I recently got the same error message while running spark application on Intellij Idea. What I did was, I downloaded the winutils.exe that is compatible with the Spark version I was running and moved it to the Spark bin directory. Then in my Intellij, I edited the configuration.

The 'Environment variables' area was empty. So, I entered HADOOP_HOME = P:\spark-2.4.7-bin-hadoop2.7 Since, the winutils.exe is in the P:\spark-2.4.7-bin-hadoop2.7\bin directory, it will locate the file while running.

So, by setting HADOOP_HOME, the null would be the HADOOP_HOME directory. Complete path would be P:\spark-2.4.7-bin-hadoop2.7\bin\winutils.exe That was how I resolved it

Failed to locate the winutils binary in the hadoop binary path

17 Answers17

Linked

Related