4

I downloaded the CDH4 tar for Hadoop with Yarn, and jobs are running fine, but I can't figure out where to view the logs from my job. In MRv1, I simply went to the JobTracker web app, and it had the job history. Individual jobs' logs were accessible from here as well, or by going to logs/userlogs directory.

In my new Yarn setup (just running on single computer), I have the logs directory, but no logs/userlogs folder.

When I go to the ResourceManager web page, localhost:8088, there is an "All Applications" section, but no entries show here. When I go to the JobHistory web page, localhost:19888, there are no jobs listed.

I'm wondering if there is a configuration issue. Here's my yarn-site.xml entries:

<property>
  <name>yarn.nodemanager.local-dirs</name> 
  <value>/Users/dolan/cdh4/_hadoop_data/yarn/local</value>
</property> 
<property> 
  <name>yarn.nodemanager.log-dirs</name>
  <value>/Users/dolan/cdh4/_hadoop_data/yarn/logs</value> 
</property>
<property> 
  <name>mapreduce.jobhistory.address</name>
  <value>localhost:10020</value> 
</property>
<property> 
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>localhost:19888</value> 
</property>
<property>
  <name>yarn.app.mapreduce.am.staging-dir</name>
  <value>/user</value>
</property>

Any tips on what to debug are greatly appreciated.

Update: I've checked the nodemanager log dir, and it is empty. Additionally, the jobhistory directories are empty. One odd thing is that Hadoop tells me, INFO mapreduce.Job: The url to track the job: http://localhost:8080/, but that address isn't available when I try from browser.

Update #2: I noticed that each job I run has the same Job ID ("job_local_0001"). This seems like a configuration issue somewhere.

Dolan Antenucci
  • 15,432
  • 17
  • 74
  • 100
  • 2
    I was able to resolve the same problem when using the hapoop-client java api. Hope this helps. http://stackoverflow.com/questions/21345022/hadoop-is-not-showing-my-job-in-the-job-tracker-even-though-it-is-running – Chris Hinshaw Jan 25 '14 at 20:02

2 Answers2

7

It sounds like the jobs here are being run locally and not on YARN at all. To make run the jobs on YARN, make sure you have this in your mapred-site.xml:

<property>
  <name> mapreduce.framework.name</name>
  <value>yarn</value>
</property>

In general, you'll probably get a faster response on questions like this by emailing the cdh-user mailing list.

Sandy Ryza
  • 71
  • 1
  • 2
2

Add following configuration into mapred-site.xml file

<property>
  <name> mapreduce.framework.name</name>
  <value>yarn</value>
</property>
<property> 
  <name>mapreduce.jobhistory.address</name>
  <value>localhost:10020</value> 
</property>
<property> 
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>localhost:19888</value> 
</property>

and remove jobhistory specific configurations as you had put in yarn-site.xml eg. above last two configs from yarn-site.xml

mehikmat
  • 80
  • 3
  • 13