Cannot initialize Cluster Hadoop-2.7.1 and YARN

Question

For simple program wordcount from examples submit job is failing even though all jobs are running.

 hadoop jar hadoop-mapreduce-examples-2.7.1.jar wordcount hdfs://localhost:9000/input hdfs://localhost:9000/output

JPS

31265 SecondaryNameNode
31064 DataNode
30929 NameNode
31478 ResourceManager
32354 Jps

error

java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
    at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1260)
    at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1256)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapreduce.Job.connect(Job.java:1255)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1284)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
    at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/usr/local/hadoop_store/hdfs/tmp</value>
  <description>A base for other temporary directories.</description>
</property>
<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
</property>
</configuration>

yarn-site.xml

<?xml version="1.0"?>
 <configuration>
<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce.shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
 <property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/datanode</value>
 </property>
</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name> mapreduce.framework.name </name>
    <value> yarn </value>
   </property>
</configuration>

I tried many solutions from others SO post like changing mapreduce.shuffle <-> mapreduce_shuffle but none of them seems working. All directories exist and webpage is also available on localhost.

System configuration:

Ubunutu:15.10
Hadoop: 2.7.1
Java: open-jdk 7u95-2.6.4-0ubuntu0.15.10.1

score 0 · Accepted Answer · answered Feb 23 '16 at 17:37

After deleting all entries from mapred-site.xml it works don't know why?

In latest 2.7.2 release mapred-site.xml.template has been changed to

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
</configuration>

This gave me a clue and I tried and it works.

score 0 · Answer 2 · answered Feb 24 '16 at 06:05

The configuration of mapred-site.xml you mentioned:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name> mapreduce.framework.name </name>
    <value> yarn </value>
   </property>
</configuration>

Here you have configured MR job execution framework to "yarn" which means using MRv2. If you want to run your job in traditional way,you can set framework to "classic" i.e. MRv1. Can you check with this configuration?

score 0 · Answer 3 · answered Apr 13 '16 at 22:51

0

delete spaces in : <name> mapreduce.framework.name </name> <value> yarn </value>

then try again

answered Apr 13 '16 at 22:51

Hicham

1

score -1 · Answer 4 · answered Feb 23 '16 at 01:46

-1

Change your yarn-site.xml parameter

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

answered Feb 23 '16 at 01:46

BruceWayne

3,286
4
25
35

try this http://stackoverflow.com/questions/19043970/cannot-initialize-cluster-please-check-your-configuration-for-mapreduce-framewo – BruceWayne Feb 23 '16 at 04:39
I have seen that too there is no such configuration I have. It's fresh installation. – sonus21 Feb 23 '16 at 05:52

Cannot initialize Cluster Hadoop-2.7.1 and YARN

4 Answers4