12

I'm trying to run a simple JUnit test with Hadoop 2.2. on my Windows 7 machine. I have an exemplary test available at http://grepalex.com/2012/10/20/hadoop-unit-testing-with-minimrcluster/ so I'm using ClusterMapReduceTestCase as a base class to start the MiniDFSCluster.

When running the test I'm getting (in console):

2013-11-15 11:59:24,636 ERROR [main] util.Shell (Shell.java:getWinUtilsPath(303)) - Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278) at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300) at org.apache.hadoop.util.Shell.(Shell.java:293) at org.apache.hadoop.util.StringUtils.(StringUtils.java:76) (...)

and in the JUnit output:

java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:435) at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996) at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:451) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:282) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200) (...)

I've read the threads at: Failed to locate the winutils binary in the hadoop binary path and Running Apache Hadoop 2.1.0 on Windows I've also had a look at the possible solutions available at http://www.srccodes.com/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os and http://www.srccodes.com/p/article/39/error-util-shell-failed-locate-winutils-binary-hadoop-binary-path

Unfortunately non of this works in case of JUnit tests. If I write junit tests I want them to be independent of the development environment (so that thye can run without problems on every developer machine). Therefore, I'm using maven and I don't want to build the required libraries on my own and put them in Hadoop bin folder, etc.

Any sugestions what would be the best solution in this case?

Community
  • 1
  • 1
Jakub
  • 3,129
  • 8
  • 44
  • 63
  • 1
    http://stackoverflow.com/questions/18630019/running-apache-hadoop-2-1-0-on-windows – constantlearner Nov 15 '13 at 11:22
  • 1
    I have already had a look at this question, however, this is not the solution to my case. I try to run MiniDFSCluster in my JUnit test and I'm using maven to manage my dependencies, therefore, I don't want to have a hadoop installation os side. I would like to embed MiniDFSCluster in my JUnit tests. – Jakub Nov 15 '13 at 12:51
  • Have the same problem here (https://issues.apache.org/jira/browse/VFS-530). And I really wonder why the mini cluster would need that native stuff. In the 1.2 days a ls.exe from cygwin was enough (and even that I guess would have been possible to ignore if one configures the "owner"). – eckes Jun 18 '14 at 01:41
  • Did you resolved the problem. Please share. Thanks – dinesh028 Oct 26 '15 at 19:11
  • I found something similar [here on this post](https://stackoverflow.com/questions/35652665/java-io-ioexception-could-not-locate-executable-null-bin-winutils-exe-in-the-ha?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa). Did you ever tried this? – Eduardo Meneses Apr 24 '18 at 01:01
  • @eckes, did you find a solution using ls.exe from cygwin? If so, could you expound on how you solved this issue? – Andrew Apr 26 '19 at 20:21
  • 1
    No sorry, did not revisit this topic. It does look like it has no prio for HDFS developers – eckes Apr 26 '19 at 21:01

1 Answers1

0

In the time since the answer was posted, Hadoop 3 has come out. However, the focus remains strongly on non-windows systems. Based on this page even the distribution of Microsoft no longer is available on Windows.

Though testing is a different matter, the consequence is that the testing for windows is not likely to receive much love, and a workaround remains required.

This answer, got accepted to answer a very similar question, the key steps:

  1. Download winutils.exe from http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe.
  2. SetUp your HADOOP_HOME environment variable on the OS level or programmatically: System.setProperty("hadoop.home.dir", "full path to the folder with winutils");

If this is still an important topic for people, note that posting on stack overflow will likely not help (if there was a nicer solution, someone likely would have posted it by now). Instead, I would recommend them to log a request on the hadoop jira , and adding a link to it in the comments here to avoid duplicates.

Dennis Jaheruddin
  • 21,208
  • 8
  • 66
  • 122