1

My target is download file from hdfs to local filesystem. I am using client which connects to remote HDFS NameNode. hadoop fs -get hdfs://sourceHDFS:8020/path_to_file/file /path_to_save_file And I got an exception.

15/03/17 12:18:49 WARN client.ShortCircuitCache: ShortCircuitCache(0x11bbad83): failed to load 1073754800_BP-703742109-127.0.0.1-1398459391664
15/03/17 12:18:49 WARN hdfs.BlockReaderFactory: I/O error constructing remote block reader.
java.io.IOException: Got error for OP_READ_BLOCK, self=/127.0.0.1:57733, remote=bigdatalite.localdomain/127.0.0.1:50010, for file /user/hive/warehouse/b2_olap_hive.db/dim_deal_log/000000_0, for pool BP-703742109-127.0.0.1-1398459391664 block 1073754800_13977

My understanding of situation. HDFS client connects to NameNode but NameNode return local DataNode IP (because NameNode and DataNode are located at the same machine). And for remote client 127.0.0.1 is wrong adress of DataNode.

How can I connect to correct datanode? And maybe my understanding is wrong?

Thanks in advance

Piotr Dajlido
  • 1,982
  • 15
  • 28
Артём Томчук
  • 21
  • 1
  • 4

1 Answers1

1

You cannot bind to 127.0.0.1. Make sure that the host name entry in /etc/hosts points to the non loopback interface. Bounce your datanode and namenode.

Pradeep Gollakota
  • 2,161
  • 16
  • 24
  • See also https://wiki.apache.org/hadoop/ConnectionRefused. But indeed, this should be written in bold in the installation documentation, everyone setting up Hadoop seems to have this issue. – bourbert May 20 '19 at 08:20