Hadoop : start-dfs.sh does not work when calling directly

Question

I have a very strange problem when starting hadoop.

When I call start-dfs.sh using absolute path /usr/local/hadoop/etc/hadoop/sbin/start-dfs.sh, it starts without any problem.

But as I add hadoop into my environment variables :

export HADOOP_HOME=/usr/local/hadoop
export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath):$CLASSPATH
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

I would like to call it directly using start-dfs.sh. But when I start like this, it throws error :

20/10/26 16:36:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on []
localhost: Error: JAVA_HOME is not set and could not be found.
localhost: Error: JAVA_HOME is not set and could not be found.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: Error: JAVA_HOME is not set and could not be found.
20/10/26 16:36:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

I wonder what is the problem ? I have all my Java home and core-site.xml well configured. Why it's not working if I start it directly from bash ?

Answer 1

It seems that you need to set the JAVA_HOME environment variable to where your java package is located in your (I suppose) Linux distribution. To do that, you have to locate the path to the java installation.

In order to do that you can use the following command on your terminal, as shown here:

find /usr -name java 2> /dev/null

which is gonna output one or a number of paths (depends on how many java versions you have on your system) like in the screenshot below:

You can choose one of the versions (or just take the single one you have) and copy the path of it.

Up next, to set the environment variable of JAVA_HOME with its path you need to copy the path you got from the output above and trim the last directory (aka the /java directory) off of it on a text editor.

For my system I chose the third version of java in my system so I went in the .bashrc file and added those 2 lines at the bottom (notice how on the setting of the variable the path ends before the /bin directory, while the setting of its path ends after the /bin directory):

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-i386
export PATH=$PATH:/usr/lib/jvm/java-11-openjdk-i386/bin/

So the bottom of the .bashrc file looks like this:

And to test it out, it works without the full path on the script (this also works for the start-all.sh and stop-all.sh scripts as well):

Answer 2

Finally the problem is that I have another hadoop in /opt/module. When I call hdfs for example, it refers to the /opt/module one than the one in /usr/local.