hadoop no data node started

Question

I am following this tutorial.

http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleCluster.html#Pseudo-Distributed_Operation

I got to this point and started the nodes.

Start NameNode daemon and DataNode daemon:

$ sbin/start-dfs.sh

But then when I run the next steps, it looks like no data node is running (as I get errors saying so).

Why is the data node down? And how can I fix this?

Here is the log from my data node.

hduser@test02:/usr/local/hadoop$ jps
3792 SecondaryNameNode
3929 Jps
3258 NameNode
hduser@test02:/usr/local/hadoop$ cat /usr/local/hadoop/logs/hadoop-hduser-datanode-test02.out
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
 -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3781
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
hduser@test02:/usr/local/hadoop$

EDIT: Seems I had this port number wrong.

<property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property>

Now when I made it right (i.e. equal to 9000) I have no name node starting up.

hduser@test02:/usr/local/hadoop$ jps
10423 DataNode
10938 Jps
10703 SecondaryNameNode

and I cannot browse:

http://my-server-name:50070/

any more.

Hope this gives you some hint what is happening.

I am total beginner with Hadoop and kind of lost now.

[core-site.xml]

<configuration>

<property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
</property>

<property>
  <name>hadoop.tmp.dir</name>
  <value>/var/lib/hadoop/tmp</value>
  <description>A base for other temporary directories.</description>
</property>

</configuration>


[hdfs-site.xml]

<configuration>

    <property>
        <name>dfs.replication</name>
        <value>1</value>
        <description>Default block replication.
          The actual number of replications can be specified when the file is created.
          The default is used if replication is not specified in create time.
        </description>
    </property>


</configuration>

In mapred-site.xml I have nothing.

I tried the accepted solution here: http://stackoverflow.com/questions/11889261/datanode-process-not-running-in-hadoop and still no luck. Any ideas or suggestions would be much appreciated. — peter.petrov, Jun 06 '14 at 12:08
@AmitDubey Do you mean system/OS wide or just for Hadoop? Hadoop already has this `export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true` — peter.petrov, Jun 06 '14 at 12:40
@AmitDubey Yes, posted. Any ideas would be much appreciated. — peter.petrov, Jun 06 '14 at 13:03
i think problem in your configuration.try this link http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/ for installation — Gabber, Jun 06 '14 at 13:11
@AmitDubey This is one of the links I was following. Could you be more specific what is wrong? I have Hadoop 2.4, this tutorial is a bit outdated. — peter.petrov, Jun 06 '14 at 13:16
sorry @peter.petrov i am using hadoop 1.2 and this tutorial worked for me many time. — Gabber, Jun 06 '14 at 13:21

score 2 · Answer 1 · edited Nov 14 '14 at 05:31

1.first stop all the entities like namenode, datanode etc. (you will be having some script or command to do that)

Format tmp directory
Go to /var/cache/hadoop-hdfs/hdfs/dfs/ and delete all the contents in the directory manually
Now format your namenode again
start all the entities then use jps command to confirm that the datanode has been started
Now run whichever application you may like or have.

Hope this helps.

Gabber · Answer 2 · 2014-06-06T13:48:55.567

Add this configuration conf/core-site.xml

<property>
<name>hadoop.tmp.dir</name>
<value>/var/lib/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>

<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
<description>The name of the default file system.  A URI whose
scheme and authority determine the FileSystem implementation.  The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class.  The uri's authority is used to
determine the host, port, etc. for a filesystem.</description> 
</property>

conf/mapred-site.xml

<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
<description>The host and port that the MapReduce job tracker runs
at.  If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>

conf/hdfs-site.xml

<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
</description>
</property>

stop hadoop

bin/stop-all.sh

change permission and remove temp directory data

chmod 755 /var/lib/hadoop/tmp
rm -Rf /var/lib/hadoop/tmp/*

format name node

bin/hadoop namenode -format

http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html — peter.petrov, Jun 06 '14 at 14:08
The `fs.default.name` is deprecated, it seems. Check my link. — peter.petrov, Jun 06 '14 at 14:08
The `mapred.job.tracker` is also deprecated. I basically tried all your suggestions before posting here. They seem come from this tutorial you told me to follow but I already followed it. — peter.petrov, Jun 06 '14 at 14:10
@peter.petrov and also try “sudo execstack –c /usr/local/hadoop/lib/native/libhadoop.so.1.0.0” — Gabber, Jun 07 '14 at 04:43

score 0 · Accepted Answer · answered Jun 07 '14 at 09:33

After 1 day of struggle, I just removed version 2.4 and installed Hadoop 2.2 (as I realized 2.2 is the latest stable version). Then I got it all working by following this nice tutorial.

http://codesfusion.blogspot.com/2013/10/setup-hadoop-2x-220-on-ubuntu.html?m=1

Something is not right with this document about 2.4 which I was reading.
Not to talk that it's not suitable for beginners, and it's usually beginners who stumble upon it.

score 0 · Answer 4 · answered Apr 10 '18 at 04:42

0

Maybe your slave's data master's data are not synced, delete data & name folder in ./hadoop/hdfs and recreate them. re-format namenode. Than start dfs.

answered Apr 10 '18 at 04:42

rahul arya

1

While this post may solve the question, including an explanation/link to original documentation really helps to improve the quality of your post. Remember that you are answering the question for readers in the future, and those people might not know the reason and/or source. – Pirate X Apr 10 '18 at 04:48

hadoop no data node started

4 Answers4