Slave nodes not in Yarn ResourceManager

Question

I've set up a 3 node Apache Hadoop cluster. On master node, I can see

[hadoop-conf]$ jps
16856 DataNode
17051 SecondaryNameNode
16701 NameNode
21601 ResourceManager
21742 NodeManager
18335 JobHistoryServer

and on slave nodes, I see

[fedora20-template dfs]$ jps
28677 Jps
28510 NodeManager
27449 DataNode

I can see three live nodes from master:50070. However, in the ResourceManager Web UI (http://master:8088/cluster/nodes), I can see only master node. Why are the two slave nodes not in the resource manager? Does that mean they are not part of resources that are used to do mapreduce?

score 17 · Accepted Answer · edited May 23 '17 at 10:30

17

Problem solved. Some configuration should be done in yarn-site.xml to let the nodemanager know where is the resource manager. Specifically, I added this property into yarn-site.xml:

  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master</value>
  </property>

Reason: the default value in yarn-default.xml is 0.0.0.0, and many properties use this hostname to contact resource manager, such as

<property>
    <name>yarn.resourcemanager.address</name>
    <value>${yarn.resourcemanager.hostname}:8032</value>
</property>

Answer credits: https://stackoverflow.com/a/22125279/3209177

edited May 23 '17 at 10:30

Community

1
1

answered Sep 23 '15 at 17:16

pythonician_plus_plus

1,244
3
15
38

I got the same error as OP, I was wondering what do you mean by `master` in the first property, should I just write master or the IPaddress? Thank you – fecke9296 Oct 27 '21 at 09:42

score 1 · Answer 2 · answered Sep 23 '15 at 01:07

1

Ideally yes, the slave nodes are not part of your cluster. Probably because of incorrect cluster setup. But just to be sure run the following command in your shell

hdfs dfsadmin -report

You should be able to get the stats of the data nodes.

answered Sep 23 '15 at 01:07

Prabhu Moorthy

178
8

I think this might be my incorrect understanding, the Hadoop Cluster, only meaning the cluster of HDFS, not yarn? So does that mean even if I have a large number of nodes, I will still have only one node to be used by yarn? Or there is some configuration that can extend the node to use? – pythonician_plus_plus Sep 23 '15 at 02:12
Your understanding seems to be wrong. A cluster does not only mean HDFS nodes. A cluster includes every node that run either a datanode daemon service or nodemanager service. In your setup the slave nodes run the Nodemanager and the Datanode daemon service. So it should ideally be part of the cluster but something seems to be wrong in the cluster configuration. Can you check if the slave nodes are given in the slaves file in hadoop conf directory on the namenode. – Prabhu Moorthy Sep 23 '15 at 06:04

score 0 · Answer 3 · answered Feb 20 '19 at 22:22

I added the following to yarn-site.xml on all nodes including NameNode (assuming it will be used as well):

<property>  
  <name>yarn.resourcemanager.resource-tracker.address</name>  
  <value>{Enter NameNode IP Address}:8025</value>  
</property>  
<property>  
  <name>yarn.resourcemanager.scheduler.address</name>  
  <value>{Enter NameNode IP Address}:8030</value>  
</property>  
<property>  
  <name>yarn.resourcemanager.address</name>  
  <value>{Enter NameNode IP Address}:8040</value>  
</property>

Slave nodes not in Yarn ResourceManager

3 Answers3

Linked