0

I am trying multi node installation of hadoop on azure server, on start-dfs.sh command by my secondarynamenode and datanode is starting but namenode fails to start with follwoing error,

2016-02-10 10:52:59,321 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Stopping services started for active state
2016-02-10 10:52:59,322 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Stopping services started for standby state
2016-02-10 10:52:59,324 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:50070
2016-02-10 10:52:59,336 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2016-02-10 10:52:59,336 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2016-02-10 10:52:59,336 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2016-02-10 10:52:59,346 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.net.BindException: Problem binding to [testhadooptest:9000] java.net.BindException: Cannot assign requested address; For more details see:  http://wiki.apache.org/hadoop/BindException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:721)
    at org.apache.hadoop.ipc.Server.bind(Server.java:425)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:574)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:2215)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:938)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:534)
    at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:509)
    at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:783)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.<init>(NameNodeRpcServer.java:344)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:673)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:646)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
Caused by: java.net.BindException: Cannot assign requested address
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.apache.hadoop.ipc.Server.bind(Server.java:408)
    ... 13 more
2016-02-10 10:52:59,349 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2016-02-10 10:52:59,356 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at testhadooptest/public_ip
************************************************************/

/etc/hosts

127.0.0.1 localhost

public_ip testhadooptest
public_ip tesdatanode

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

But when I comment line public_ip testhadooptest from /etc/hosts, namenode start, why this strange behavior?. I check link and link, but did not find a solution. Please help.

Hadoop files:

core-site.xml

<configuration>
   <property>
      <name>fs.default.name</name>
      <value>hdfs://testhadooptest:9000</value>
   </property>
</configuration>

hdfs-site.xml

<configuration>
    <property>
       <name>dfs.replication</name>
       <value>1</value>
    </property>
    <property>
       <name>dfs.namenode.name.dir</name>
       <value>file:/home/hduser/mydata/hdfs/namenode</value>
    </property>
    <property>
       <name>dfs.datanode.data.dir</name>
       <value>file:/home/hduser/mydata/hdfs/datanode</value>
    </property>
</configuration>

Hadoop Verion 2.7.1

Thanks in Advance :)

Community
  • 1
  • 1
Prashant Puri
  • 2,324
  • 1
  • 15
  • 21

1 Answers1

1

Problem was with azure server, it did not allow to bind on PUBLIC VIRTUAL IP (VIP) ADDRESS, it just allow binding on INTERNAL IP ADDRESS, so I create virtual network, add all 3 nodes to the virtual network.

Also replace public_ip to internal_ip in /etc/hosts on every machine

New /etc/hosts 

127.0.0.1 localhost

internal_ip testhadooptest
internal_ip tesdatanode

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
Prashant Puri
  • 2,324
  • 1
  • 15
  • 21