UnsatisfiedLinkError when running Hadoop MapReduce Java program

Question

I am trying to run this Map Reduce program with Hadoop on Windows 8.1. After much effort, I have gotten it pretty close to working. I've got Java 1.8.0_45 and Hadoop-2.7.0. I also have the winutils.exe and hadoop.dll which have caused issues for a lot of people.

Here is the code:

public class OSProject {

public static class Map extends MapReduceBase implements
        Mapper<LongWritable, Text, Text, IntWritable> {

    @Override
    public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {

        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);

        while (tokenizer.hasMoreTokens()) {
            value.set(tokenizer.nextToken());
            output.collect(value, new IntWritable(1));
        }

    }
}

public static class Reduce extends MapReduceBase implements
        Reducer<Text, IntWritable, Text, IntWritable> {

    @Override
    public void reduce(Text key, Iterator<IntWritable> values,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        int sum = 0;
        while (values.hasNext()) {
            sum += values.next().get();
        }

        output.collect(key, new IntWritable(sum));
    }
}

public static void main(String[] args) throws Exception {

    BasicConfigurator.configure();
    JobConf conf = new JobConf(OSProject.class);
    conf.setJobName("wordcount");

    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);

    conf.setMapperClass(Map.class);
    conf.setReducerClass(Reduce.class);

    conf.setInputFormat(TextInputFormat.class);
    conf.setOutputFormat(TextOutputFormat.class);

    FileInputFormat.setInputPaths(conf, new Path("c:/hwork/input"));
    FileOutputFormat.setOutputPath(conf, new Path("c:/hwork/output"));

    //FileInputFormat.setInputPaths(conf, new Path(args[0]));
    //FileOutputFormat.setOutputPath(conf, new Path(args[1]));

    JobClient.runJob(conf);

}
}

The problem is that the program throws this error when I run it:

Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)
at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:473)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:526)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:504)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:305)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:147)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:833)
at osproject.OSProject.main(OSProject.java:86)

The line that it errors on is:

JobClient.runJob(conf);

What it looks like is that it cannot create the output file for some reason. Any answer as to why this is happening and how to fix it would be greatly appreciated.

I guess you build this app with JDK 1.8. You should check if you run the app with proper JRE, maybe your default JRE is from Java 6 — user1723095, May 10 '15 at 20:04
Are your paths ok? Please double check "c:/hwork/input" or use relative path for now — user1723095, May 10 '15 at 20:45
I am not seeing any issue with the input path. With the output path, if the folder exists, it gives an error stating that "Output directory file: /hwork/output already exists". So I assume one of the methods in the libraries is supposed to create the directory itself. It gives the error specified in the question if the directory "C:/hwork/output" does not exist. I have tried setting it to a number of different paths, but it has no effect. — Moatimus, May 10 '15 at 21:32
possible duplicate of [Hadoop on Windows. YARN fails to start with java.lang.UnsatisfiedLinkError](http://stackoverflow.com/questions/30964216/hadoop-on-windows-yarn-fails-to-start-with-java-lang-unsatisfiedlinkerror) — centic, Sep 04 '15 at 05:56

score 1 · Answer 1 · edited May 23 '17 at 12:14

1

UnsatisfiedLinkError is thrown when some lib couldn't be found (or it was found but doesn't contain implementation of some function) and thus native method can't be found. In my opinion your app can't find proper lib.

Your problem is similar to this one: hadoop mapreduce: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

So I would suggest runnig your app with LD_LIBRARY_PATH pointing to directory which contains hadoop.dll.

edited May 23 '17 at 12:14

Community

1
1

answered May 11 '15 at 11:08

user1723095

1,181
1
13
24

Amazing! My peer's 32 bit machine works with your suggested `LD_LIBRARY_PATH`, thank you! – tkhm Sep 07 '17 at 09:23

UnsatisfiedLinkError when running Hadoop MapReduce Java program

1 Answers1