5

I have two VMs setup with Ubuntu 12.04. I am trying to setup Hadoop multinode, but after executing hadoop/sbin/start-dfs.shI see following process on my master:

20612 DataNode
20404 NameNode
20889 SecondaryNameNode
21372 Jps

However, there is nothing in the slave. Also when I do hdfs dfsadmin -report, I only see:

Live datanodes (1):

Name: 10.222.208.221:9866 (master)
Hostname: master

I checked logs, my start-dfs.sh does not even try to start datanode on my slave. I am using following configuration:

#/etc/hosts
127.0.0.1       localhost
10.222.208.221  master
10.222.208.68   slave-1

changed hostanme in /etc/hostname in respective systems Also, I am able to ping slave-1 from master system and vice-versa using ping.

/hadoop/etc/hadoop/core-site.xml

<configuration>
   <property>
      <name>fs.defaultFS</name>
      <value>hdfs://master:9000</value>
   </property>
</configuration>

#hadoop/etc/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///hadoop/data/namenode</value>
        <description>NameNode directory</description>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///hadoop/data/datanode</value>
        <description>DataNode directory</description>
    </property>

    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
 </configuration>

/hadoop/etc/hadoop/mapred-site.xml

<configuration>
   <property>
         <name>mapred.job.tracker</name>
         <value>master:9001</value>
   </property>
</configuration>

I have also added master and slave-1 in /hadoop/etc/master and /hadoop/etc/slaveson both my master and slave system.

I have also tried cleaning data/* and then hdfs namenode -format before start-dfs.sh, still the problem persists.

Also, I have Network adapter setting marked as Bridged adapter.

Any possible reason datanode not starting on slave?

Ritu Raj
  • 543
  • 2
  • 6
  • 23
  • `mapred.job.tracker` doesn't exist in Hadoop2, however that's not a DFS problem... Can you add your log output to the question? Also are you sure you want 3 replicas on 2 datanodes? FWIW, Apache Ambari has an automated Hadoop installer – OneCricketeer Apr 22 '18 at 05:10
  • I've been beating my head on this all day. Would love an answer. I have one more datanode than the OP, but otherwise my config looks nearly identical. No idea why start-dfs.sh is starting a datanode on my namenode. – tadamhicks Jun 25 '18 at 21:37

2 Answers2

4

Can't claim to have the answer, but I found this "start-all.sh" and "start-dfs.sh" from master node do not start the slave node services?

Changed my slaves file to workers file and everything clicked in.

tadamhicks
  • 905
  • 1
  • 14
  • 34
1

It seems you are using hadoop-2.x.x or above, so, try this configuration. And by default masters file( hadoop-2.x.x/etc/hadoop/masters) won't available on hadoop-2.x.x onwards.

hadoop-2.x.x/etc/hadoop/core-site.xml:

<configuration>
 <property>
 <name>fs.default.name</name>
  <value>hdfs://master:9000</value>
 </property>
</configuration>

~/etc/hadoop/hdfs-site.xml:

<configuration>
<property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///hadoop/data/namenode</value>
    <description>NameNode directory</description>
</property>
<property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///hadoop/data/datanode</value>
    <description>DataNode directory</description>
</property>

<property>
    <name>dfs.replication</name>
    <value>2</value>
</property>
</configuration>

~/etc/hadoop/mapred-site.xml:

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>

~/etc/hadoop/yarn-site.xml:

<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>master</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>

<property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

~/etc/hadoop/slaves

slave-1

copy all the above configured file from master and replace it on slave on this path hadoop-2.x.x/etc/hadoop/.

ArunTnp
  • 74
  • 8
  • Thank you for your solution. I did not realize that the "workers" was called "slaves" in hadoop 2.x. – Herry Aug 02 '21 at 13:35