Stack Overflow
Stack Overflow

Questions tagged [datanode]

A DataNode stores data in the HFS (HadoopFileSystem). A functional filesystem has more than one DataNode, with data replicated across them.

86 questions
18
votes
14 answers

localhost: ERROR: Cannot set priority of datanode process 32156

I am trying to install hadoop on ubuntu 16.04 but while starting the hadoop it will give me following error localhost: ERROR: Cannot set priority of datanode process 32156. Starting secondary namenodes [it-OptiPlex-3020] 2017-09-18 21:13:48,343…
Shubham Hupare
  • 321
  • 1
  • 3
  • 11
4
votes
1 answer

Apache Hadoop Windows 10 - Datanode, Resource Manager and Yarn immediately shutdown on startup

I have followed a tutorial to setup Apache Hadoop for Windows, which can be found here. I am now having an issue with the Datanode, Resource Manager, and Yarn cmd windows showing that all 3 shutdown seconds after opening, with only the Namenode…
Tom Hood
  • 497
  • 7
  • 16
3
votes
1 answer

Google Dataproc writing from a client app uses cluster's internal IP for Datanodes

I have setup a Dataproc cluster on Google Cloud. It is sup and running and I can access HDFS and copy files from the SSH 'in browser" console. So the problem is not on the Dataproc side. I am now using Pentaho (ELT software) to copy files. Pentaho…
Jean-François Monteil-Lemoine
  • 102
  • 7
3
votes
2 answers

ERROR in datanode execution while running Hadoop first time in Windows 10

I am trying to run Hadoop 3.1.1 in my Windows 10 machine. I modified all the files: hdfs-site.xml mapred-site.xml core-site.xml yarn-site.xml Then, I executed the following command: C:\hadoop-3.1.1\bin> hdfs namenode -format The format ran…
Chintan Rajvir
  • 689
  • 6
  • 20
3
votes
0 answers

AWS EMR - How to extend hdfs capacity

Our cluster is running with 2 core nodes with little dfs capacity and it needs to be increased. I added a new volume of 500GB to the core node instance and mounted it to /mnt1 and updated the hdfs-site.xml in both master and core nodes. …
Kannan AnandaKrishnan
  • 161
  • 5
2
votes
0 answers

Datanode is not getting started at Windows 10 for Hadoop 3.1.3

I am trying bring up datanode and namenode for hadoop 3.1.3 on windows 10 ,and I have kept the required winutils.exe and hadoop.dll in bin folder as well as in system 32 folder .But still I am getting below exception for datanode : 2021-04-09…
deb
  • 21
  • 1
2
votes
2 answers

Hadoop Can't access datanode without using the IP

I have the following system: Windows host Linux guest with Docker (in Virtual Box) I have installed HDFS in Docker (Ubuntu, Virtual Box). I have used the bde2020 hadoop image from Docker Hub. This is my docker-compose: namenode: image:…
David Zamora
  • 383
  • 1
  • 4
  • 15
2
votes
0 answers

HDFS NameNode startup very slow with few blocks

I have a fairly small setup (HDP 2.6) with roughly 1429 blocks on a 15 TB HDD. The system has 512 GB RAM and 128 cores (256 threads). Over last few days, I've seen the startup of entire HDP setup go from about 10 minutes to several hours. The…
MojoJojo
  • 3,897
  • 4
  • 28
  • 54
2
votes
0 answers

Connecting to HDFS namenode running in docker container from outside host VM

I have a HBase + HDFS setup, in which each of the HBase master, regionservers, HDFS namenode and datanodes are containerized. When running all of these containers on a single host VM, things work fine as I can use the docker container names…
anthr
  • 1,026
  • 4
  • 17
  • 34
2
votes
1 answer

Do we have to format datanode for HDFS file system or we only need to format namenode

While setting up pseudo or full cluster, do we need to format data node to install HDFS file system or we only have to format namenode. I am asking this because, I have read this on many blogs that, HDFS stores the data in sequential order on the…
Sagar Chawla
  • 59
  • 3
2
votes
3 answers

Hadoop start-all.cmd command : datanode shutting down

I am trying to install hadoop in my windows 10 Reference : https://github.com/MuhammadBilalYar/Hadoop-On-Window/wiki/Step-by-step-Hadoop-2.8.0-installation-on-Window-10 Hadoop start-all.cmd command start namenode , resourceManager and nodeManager…
Bharti Ladumor
  • 1,624
  • 1
  • 10
  • 17
2
votes
2 answers

how to tune the "DataNode maximum Java heap size" in hadoop clusters

I searched in google to find info about how to tune the value for - DataNode maximum Java heap size ,except this one…
Judy
  • 1,595
  • 6
  • 19
  • 41
2
votes
0 answers

Hadoop data node shutting down with message 'libhadoop cannot be loaded'

I'm trying to start datanode on new slaves in order to add them to a live hadoop cluster. But failing to start datanode with command hadoop-daemon.sh start datanode I have created file /var/lib/hadoop-hdfs/dn_socket manually and also changed its…
akshay naidu
  • 115
  • 4
  • 18
1
vote
1 answer

writing to hdfs error: Not enough replicas was chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1}

I encountered the problem similar to this: https://stackoverflow.com/questions/52809233/failed-to-place-enough-replicas-expected-size-is-1-but-only-0-storage-types-can my name node logs in $HADOOP_HOME/logs/ while the storage policy is set to…
mehran
  • 39
  • 3
1
vote
0 answers

Is there a way to get DataNode's DiskUsage using java client?

I want to get disk-usage status of a DataNode over RPC/ Java Client. I know there is a way using WebAPI and parsing the response. But, I am looking for RPC based approach only.
dip_tech
  • 53
  • 5
1
2 3 4 5 6