Cannot initialize cluster exception while running job on Hadoop 2

Question

The question is linked to my previous question All the daemons are running, jps shows:

6663 JobHistoryServer
7213 ResourceManager
9235 Jps
6289 DataNode
6200 NameNode
7420 NodeManager

but the wordcount example keeps on failing with the following exception:

ERROR security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
    at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1238)
    at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1234)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
    at org.apache.hadoop.mapreduce.Job.connect(Job.java:1233)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1262)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
    at WordCount.main(WordCount.java:80)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Since it says the problem is in configuration, I am posting the configuration files here. The intention is to create a single node cluster.

yarn-site.xml

<?xml version="1.0"?>
 <configuration>
<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

core-site.xml

<configuration>
<property>
   <name>fs.default.name</name>
   <value>hdfs://localhost:9000</value>
</property>
</configuration>

hdfs-site.xml

 <configuration>
 <property>
   <name>dfs.replication</name>
   <value>1</value>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/home/hduser/yarn/yarn_data/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/home/hduser/yarn/yarn_data/hdfs/datanode</value>
 </property>
 </configuration>

mapred-site.xml

<configuration>
  <property>
      <name>mapreduce.framework.name</name>
      <value>Yarn</value>
  </property>
</configuration>

Please tell what is missing or what am I doing wrong.

neeraj · Answer 1 · 2014-06-19T06:53:12.390

21

I was having similar issue, but yarn was not the issue. After adding following jars into my classpath issue got resolved:

hadoop-mapreduce-client-jobclient-2.2.0.2.0.6.0-76
hadoop-mapreduce-client-common-2.2.0.2.0.6.0-76
hadoop-mapreduce-client-shuffle-2.2.0.2.0.6.0-76

edited Jun 19 '14 at 06:53

answered May 30 '14 at 13:40

neeraj

890
2
15
22

Did you add the jars using CLASSPATH variable or libjars ? – Pratik Khadloya Aug 21 '14 at 17:13
just add those lib into yours build path i.e classpath – neeraj Sep 01 '14 at 08:30
2

If you are using Maven or some other build tool, it's sufficient to specify `hadoop-mapreduce-client-jobclient` as the only dependency from this list. Both `common` and `shuffle` are `jobclient`'s dependencies, so you don't have to specify them as your project's dependencies. – falconepl Dec 09 '14 at 17:31
@bat_rock, i had the same issue. your solution worked for me :-) – Spike Aug 27 '16 at 08:43
1

@neeraj How to add jar files to classpath? – Jon Andrews Jul 06 '17 at 09:42

score 4 · Accepted Answer · answered Oct 28 '13 at 19:07

4

You have uppercased Yarn, which is probably why it can not resolve it. Try the lowercase version that is suggested in the official documentation.

<configuration>
  <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
  </property>
</configuration>

answered Oct 28 '13 at 19:07

Thomas Jungblut

20,854
6
68
91

It worked, changing Yarn to yarn solved the problem and the job ran successfully! Thanks! – SpeedBirdNine Oct 28 '13 at 19:36
Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses. even my configuration is OK. – Shyam Gupta Jul 01 '17 at 11:48

score 2 · Answer 3 · edited May 23 '17 at 11:45

Looks like i had a lucky day and went with this exception through 'all' of those causes. Summary:

wrong mapreduce.framework.name (see above)
missing mapreduce job-client jars (see above)
wrong version (see Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses-submiting job2remoteClustr )
my configured 'yarn.ipc.client.factory.class' wasn't in the classpath of the yarn server's (just on the client)

score 1 · Answer 4 · answered Aug 21 '14 at 21:52

In my case i was trying to use sqoop and ran into this error. Turns out that i was pointing to the latest version of hadoop 2.0 available from CDH repo for which sqoop was not supported. The version of cloudera was 2.0.0-cdh4.4.0 which had yarn support build in.

When i used 2.0.0-cdh4.4.0 under hadoop-0.20 the problem went away.
Hope this helps.

score 0 · Answer 5 · answered Sep 15 '14 at 19:29

0

changing mapreduce_shuffle to mapreduce.shuffle made it work in my case

answered Sep 15 '14 at 19:29

alex

1,757
4
21
32

Cannot initialize cluster exception while running job on Hadoop 2

5 Answers5

Linked