2

I installed Hadoop and YARN on my MAC. I am able to run the wordcount example and output results on HDFS (pesudo-distribute mode), I know my program is running data file on HDFS because I have to copy file to HDFS for it to run. However, when I check on my WEB UI for YARN, there is no record that any task has been accomplished. After research online, it seems that the application is still running locally.

After start-yarn.sh, I can open Resource Manager web UI http://localhost:8088/, problem is that no task ever shows up. Furthermore, I cannot even access to my JobTracker Web UI.

I found out the following link. it has a similar problem with mine, but the solution doesn't work for me

Hadoop is not showing my job in the job tracker even though it is running

My XML set-up is as follows

mapred-site.xml

<configuration>
  <property>
      <name>mapred.job.tracker</name>
      <value>localhost:9001</value>
  </property>
  <property>
    <name>mapred.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

yarn-site.xml

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
    <name>yarn.resourcemanager.address</name>
    <value>localhost:9002</value>
</property>

core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

Anyone knows what the problem is? Thanks

Community
  • 1
  • 1
SY Z
  • 53
  • 3
  • 10

1 Answers1

1

If you are using yarn as there is not jobtracker, it will be resourcemanager which will handle you request. So as you have given mapred.framework.service as yarn, so framework will run resourcemanager. remove the entry for jobtracker from mapred-site.xml try with below mapred-site.xml

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>localhost:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>localhost:19888</value>
</property>

also start history server by bin/yarn-damemoe.sh start historyserver
by this you will be able to watch the history of your finished application(job in old mapreduce). On web UI you will only show the running tasks, for more detail it gives link to application master which have all the responsibility to run the application in yarn. also after finish task you can wath history by click link to history. try to find ll these link on resourmanager web UI

agarwal_achhnera
  • 2,582
  • 9
  • 55
  • 84
  • Thank you for the suggestion. I started the history server and changed mapred-site.xml as you pointed out. Do you mean I will not find any task on the Application->Finished tab? Then how do I find performance evaluation for any tasks I had run on hadoop? And I can't find link to histroy, would you mind point it? Thank you. – SY Z Sep 22 '14 at 03:48