1

I started with the need to backup the whole hadoop datanode data directory using:

hdfs dfs -copyToLocal /var/hadoop/dfs/name/data /home/ubuntu/hadoopfiles

And I got an error:

"No such file opr directory" for /var/hadoop/dfs/name/data

After some search I've found this Stack Overflow topic with examples: https://stackoverflow.com/questions/28213116/hadoop-copy-a-local-file-system-folder-to-hdfs#=

But even when I do:

hdfs dfs -ls

I receive the error message ls: `.': No such file or directory

I have also looked to other posts, it seems this is a quite common issue, but I was unable to find a solution for me.

Thanks.

Giovanni Tusa
  • 11
  • 1
  • 3
  • Could you please remove that `./ `? I don't understand why you putting `./` before `hdfs dfs` ? – Vin Mar 20 '19 at 14:56

1 Answers1

2

First, use 

hadoop fs -get /theFolder

 to copy it into the current directory you are ssh'ed into on your box.

Then you can use either scp or my preference of rsync to copy the files between your box and your local system like so. Here's how I'd use rsync after having used the -get, still in the same directory:

rsync -av ./theFolder username@yourlocalmachine:/home/username

This will copy theFolder from the local fs on your box into your home folder on your machine's fs. Be sure to replace username with your actual username in both cases, and yourlocalmachine with your machine's hostname or ip address.

Pls refer this answer

Copy from Hadoop to local machine

Or look this link scp method

Arun Solomon
  • 421
  • 3
  • 13
  • thanks for the answer. There was a misunterstanding on the real issue I am facing. I don't have any problem on transfering data from one machine to another. The point is that if I follow the documentation of "hdfs dfs" or "hadoop fs" commands, it seems that the (hadoop datanode directory) from where I wish to get data, is never found with "No such file or directory" error. I am trying to copy to the local filesystem of the same hadoop datanode server. From your answer, it seems I don't need any param in the "hadoop fs -get" command, just the . Correct? – Giovanni Tusa Mar 22 '19 at 08:50
  • I read more carefully your answer. Actually in your example the (hadoop data) is /theFolder and the destination is the current dir. This is not working for me. As I wrote in the original question, "hadoop fs /theFolder ." will give me "No such file or directory" for "theFolder". – Giovanni Tusa Mar 22 '19 at 08:56