6

I am using Hadoop2.2. I see that my jobs are completed with success. I can browse the filesystem to find the output. However, when I browse http://NNode:8088/cluster/apps, I am unable to see any applications that have been completed so far ( I ran 3 wordcount jobs, but none of it is seen here).

Are there any configurations that need to be taken into account?

enter image description here

Here is the yarn-site.xml

<property>
        <name>yarn.resourcemanager.hostname</name>
        <value>NNode</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
<!--
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
-->

Here is mapred-site.xml:

<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

I have job history server running too:

jps
4422 NameNode
5452 Jps
4695 SecondaryNameNode
4924 ResourceManager
72802 Jps
5369 JobHistoryServer
brain storm
  • 30,124
  • 69
  • 225
  • 393
  • Try to check hadoop logs files. you might find warnings or errors there. But it seems that your datanode is down – eliasah Jul 01 '14 at 20:06
  • @eliasah: The datanode is running fine. I did not show `jps` on datanode. what I showed above is namenode(master). The datanode runs fine and even my jobs are finished. – brain storm Jul 01 '14 at 20:11
  • @brainstorm Can you post the entire output of a wordcount job? All of it. – Mike Park Jul 09 '14 at 19:50
  • And by output, I mean what you get in the terminal – Mike Park Jul 09 '14 at 19:50
  • @climbage: Thanks for helping out. Two things I observed, one is I need to start yarn in my namenode. Then it seems to work. If I do `start-yarn.sh` in my client, it does not start resourcemanager. I dont know why. Secondly, Although I see my job being listed in application window, I cannot find information about number of mappers and reducers etc. In hadoop1, it was easy to find them in 50030 port I guess. Do you know how to get that info, in hadoop2 on the UI – brain storm Jul 09 '14 at 20:44

2 Answers2

0

After applications are completed, their responsibility might be moved to Job History Server. So check Job History Server URL. It normally listen on port 19888. E.g.

http://<job_history_server_address>:19888/jobhistory

Log directories and log retain durations are configurable in yarn-site.xml. With YARN, even one can aggregate logs to a single (configurable) location.

Sometimes, even though application is listed, logs are not available (I am not sure if its due to some bug in YARN). However, almost each time I was able to get the logs using command line:

yarn logs -applicationId the_application_id

Athough there are multiple options. Use help for details:

yarn logs --help

Vasu
  • 4,862
  • 8
  • 42
  • 48
0

you can refer Hadoop is not showing my job in the job tracker even though it is running

                    conf.set("fs.defaultFS", "hdfs://master:9000");
                    conf.set("mapreduce.jobtracker.address", "master:54311");
                    conf.set("mapreduce.framework.name", "yarn");
                    conf.set("yarn.resourcemanager.address", "master:8032");

I tested in my cluster. It works!

Community
  • 1
  • 1