Trying to get Hadoop 3.2.2 up and running on Ubuntu 18.
Followed this tutorial. https://phoenixnap.com/kb/install-hadoop-ubuntu
Tried to troubleshoot, I read this thread: Datanode process not running in Hadoop
Examined Datanode log, saw permissions issue.
Because Part 5 of the 12-vote answer in the above-linked thread seemed the most relevant, I did this:
cd dfsdata
sudo chmod -R 755 datanode
cd ..
cd hadoop-3.2.2
cd sbin
./stop-all.sh
hadoop namenode -format
start-all.sh
jps
But still no DataNode in the list.
(This was slightly out of order from the suggested process; I did not stop-all before changing permissions, does that matter?)
Datanode log file:
************************************************************/
2021-06-02 15:14:24,546 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
2021-06-02 15:14:25,607 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/home/hdoop/dfsdata/datanode
2021-06-02 15:14:25,737 WARN org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker: Exception checking StorageLocation [DISK]file:/home/hdoop/dfsdata/datanode
EPERM: Operation not permitted
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:382)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:863)
at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:510)
at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:491)
at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:513)
at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:234)
at org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:141)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:116)
at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:239)
at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:52)
at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:142)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-06-02 15:14:25,740 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, volumes failed: 1, volume failures tolerated: 0
at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:231)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2806)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2721)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2763)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2907)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2931)
2021-06-02 15:14:25,742 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 0, volumes configured: 1, volumes failed: 1, volume failures tolerated: 0
2021-06-02 15:14:25,760 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at ip-172-31-94-192.ec2.internal/172.31.94.192
************************************************************/
Previous threads are a little confusing to me because they indicate different folder names and things like that, probably due to using previous versions of hadoop and/or different setup processes.
EDIT: Also tried doing stop-all, changing permissions on both datanode and namenode, and start-all. Didn't seem to change anything.
EDIT 2: This thread actually looks very similar: Hadoop datanode not running
