Yarn slave nodes are not communicating with master node?

Question

I am not able to see my nodes when I do yarn node -list, even though I have configured /etc/hadoop/conf/yarn-site.xml with the correct properties (it seems to me, at least according to this question Slave nodes not in Yarn ResourceManager).

Here's what I've done so far:

installed resourcemanager on the master
installed nodemanager on the slaves
checked yarn-site.xml for this on ALL the nodes:

<property> <name>yarn.resourcemanager.hostname</name> <value>master-node</value> </property>
after modifying the config file, restarted resourcemanager and nodemanager on the master and slaves, respectively.

But yet when I do yarn node -list I only see

Total Nodes: 0 
Node-Id       Node-state    Node-Http-Address      Number-of-Running-Containers

At my nodes, I looked at the .out files in /var/log/hadoop-yarn/ and I see this in them:

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 244592
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

EDIT: when I look at the .log files I see the following, but I'm not sure how to fix it:

    INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state STARTED; cause:

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: <master node ip>:8020:8031 (configuration property 'yarn.resourcemanager.resource-tracker.address')

Caused by: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: <master node ip>:8020:8031 (configuration property 'yarn.resourcemanager.resource-tracker.address')

How do I connect my slave nodes to my master node?

did you setup password less ssh between master and slave if not refer this link to do the setup http://www.tecmint.com/ssh-passwordless-login-using-ssh-keygen-in-5-easy-steps/ — Bhavesh, Mar 23 '17 at 07:26
No. This is not vanilla installation. Password less ssh is not required. Does the slave nodes have their `/etc/hosts` configured to resolve hostnames? — franklinsijo, Mar 23 '17 at 08:51
@Sother Do you have `master-node` as value in the property files or its actual IP address? If it is `master-node`, the IP address of the node must be mapped with its hostname in `/etc/hosts` file — franklinsijo, Mar 23 '17 at 09:15
no, I do not literally have "`master-node`". I have the actual ip address. — makansij, Mar 23 '17 at 09:26
Check the `nodemanager` logs under `/var/log/hadoop-yarn/` directory in the slave node. — franklinsijo, Mar 23 '17 at 09:45
There is more to it @Sother. Scroll down to the end and look out for `ERROR` messages — franklinsijo, Mar 23 '17 at 17:13
you were right...sorry it is deceiving. I updated the question. thank you. — makansij, Mar 23 '17 at 17:30

score 3 · Answer 1 · answered Mar 23 '17 at 18:16

The value set for yarn.resourcemanager.hostname acts as the base value for all the ResourceManager properties. The property yarn.resourcemanager.resource-tracker.address defaults to the value of ${yarn.resourcemanager.hostname}:8031. Refer yarn-default.xml for the complete list of default YARN configurations.

And from the nodemanager ERROR log,

Caused by: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: <master node ip>:8020:8031 (configuration property 'yarn.resourcemanager.resource-tracker.address')

It looks the yarn.resourcemanager.hostname property is configured incorrectly as <master node ip>:8020 instead of <master node ip> on the slave nodes.

Edit the yarn-site.xml on all the nodes to have

<property>
   <name>yarn.resourcemanager.hostname</name>
   <value>master_node</value> <!-- IP address or Hostname of the node where Resource Manager is started, Omit the port number -->
</property>

Finally, restart the YARN services.

score 1 · Answer 2 · answered Mar 23 '17 at 08:37

1

please set all this properties and try
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>master_node:8032</value>
      </property>
      <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>master_node:8033</value>
      </property>
      <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>master_node:8030</value>
      </property>
      <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>master_node:8031</value>
      </property>
      <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>master_node:8088</value>
      </property>
      <property>
        <name>yarn.resourcemanager.webapp.https.address</name>
        <value>master_node:8090</value>
      </property>

answered Mar 23 '17 at 08:37

Kanagaraj Dhanapal

451
2
10

1

This is redundant. Setting `yarn.resourcemanager.hostname` is sufficient. All these properties use the value set for `yarn.resourcemanager.hostname` as the base value. – franklinsijo Mar 23 '17 at 08:43
Do I need the port number? I just have the ip – makansij Mar 23 '17 at 08:45
If you have not set it anywhere, it will be using the default port number which is `8032`. Unless you wish to change the port, it is not necessary – franklinsijo Mar 23 '17 at 08:48
i guess we have to set all these properties. please refer this link : https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-common/yarn-default.xml – Kanagaraj Dhanapal Mar 23 '17 at 09:14
No, that is pretty evident from the link you have provided. – franklinsijo Mar 23 '17 at 09:18
ok buddy. but i have doubt. that is they are referring the value of yarn.resourcemanager.hostname instead of mentioning all the properties. – Kanagaraj Dhanapal Mar 23 '17 at 09:21
Like when we have case to change the value of the property. simply we can change one place and it reflect to all the properties – Kanagaraj Dhanapal Mar 23 '17 at 09:23
My final question buddy. do we want to set all these properties are not? – Kanagaraj Dhanapal Mar 23 '17 at 09:24
if not, please go through this link : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html – Kanagaraj Dhanapal Mar 23 '17 at 09:41
No, it is not working. I'm confused from the information above. Can each of you please summarize in an individual answer. @Kanagarajdhanapal you can just edit your answer please. – makansij Mar 23 '17 at 15:51

score 1 · Answer 3 · edited Jun 20 '20 at 09:12

You need to set ip for yarn.resourcemanager.hostname property. if you want to use the hostname, your machine needs to know to which ip that hostname is pointing to. So you need to add host entry in /etc/hosts file.

To do that,

Open terminal
Type vim /etc/hosts and hit enter
Add this line at end of the file (use key i to enable insertion)

<your resourcemanager ip><space><your hostname>
```
example: `192.168.1.23 master-node`
```
Save the file by typing <Esc> + :wq
Restart the nodemanager

I recommend using ambari kind of managing tool to do these kind of stuffs. This allows easy modification of configuration at any time for hadoop environment. Because manual work always have more chance for error.

Yarn slave nodes are not communicating with master node?

3 Answers3