How to copy file from HDFS to the local file system

Question

How to copy file from HDFS to the local file system . There is no physical location of a file under the file , not even directory . how can i moved them to my local for further validations.i am tried through winscp .

score 287 · Accepted Answer · edited Jan 14 '16 at 19:09

287

bin/hadoop fs -get /hdfs/source/path /localfs/destination/path
bin/hadoop fs -copyToLocal /hdfs/source/path /localfs/destination/path
Point your web browser to HDFS WEBUI(namenode_machine:50070), browse to the file you intend to copy, scroll down the page and click on download the file.

edited Jan 14 '16 at 19:09

VeLKerr

2,995
3
24
47

answered Jul 24 '13 at 15:11

Tariq

34,076
8
57
79

perfect tariq , i got the it ,**There is no physical location of a file under the file , not even directory .** bin/hadoop dfs -ls /use/hadoop/myfolder i can view the file , From i got the info as **To inspect the file, you can copy it from HDFS to the local file system** , so i though i can moved them from winscp – Surya Jul 24 '13 at 15:25
2

once again i need to mention tariq , thanks a lot for contributing you time and knowledge . thanks a lot . u did support a lot , this gives a lot of confidence for a new bie like me . – Surya Jul 24 '13 at 15:27
1

I see. You can actually use hdfs cat command if you wish to see the file's content or open the file on the webui. This will save you from downloading the file to your local fs. You are welcome. And if you are 100% satisfied with the answers to your questions you can mark them so that others can benefit from it.. Not for just this one, but in general. – Tariq Jul 24 '13 at 15:38
2

Just to add to my lat comment, if it is a binary file, cat won't show you the actual content. To view the content of a binary file you can use : bin/hadoop fs -text /path/to/file – Tariq Jul 24 '13 at 16:18
i tried using xml file , http://stackoverflow.com/q/17851462/2499617 , got Diskerror on slave can you please advise – Surya Jul 25 '13 at 07:26
1

It seems to be a bug(fixed). See the answer. – Tariq Jul 25 '13 at 08:13
It there a possibility to specify the modification/creation date of the files your copy? – marlieg Apr 20 '15 at 16:30

Ani Menon · Answer 2 · 2016-08-08T09:52:11.040

40

In Hadoop 2.0,

hdfs dfs -copyToLocal <hdfs_input_file_path> <output_path>

where,

hdfs_input_file_path maybe obtained from http://<<name_node_ip>>:50070/explorer.html
output_path is the local path of the file, where the file is to be copied to.
you may also use get in place of copyToLocal.

edited Aug 08 '16 at 09:52

answered Aug 08 '16 at 09:20

Ani Menon

27,209
16
105
126

score 25 · Answer 3 · edited Jul 26 '21 at 07:24

25

In order to copy files from HDFS to the local file system the following command could be run:

hadoop dfs -copyToLocal <input> <output>

<input>: the HDFS directory path (e.g /mydata) that you want to copy
<output>: the destination directory path (e.g. ~/Documents)

Update: Hadoop is deprecated in Hadoop 3

use hdfs dfs -copyToLocal <input> <output>

edited Jul 26 '21 at 07:24

CanCoder

1,073
14
20

answered Sep 16 '14 at 10:45

Hafiz Muhammad Shafiq

8,168
12
63
121

2

This does not seem to work for me. It always says file not found. I am using cloudera's VM instance which has cent os 6.4 – SutharMonil Nov 30 '15 at 11:49
@SutharMonil Are you sure the file is actually there? Can you browse there via `hadoop fs -ls` ? – Dennis Jaheruddin Dec 06 '15 at 21:34
Just use `hadoop dfs -get `, it'll work. To list files, use `hadoop dfs -ls ` – jbaranski May 12 '22 at 13:16

score 7 · Answer 4 · answered Feb 06 '17 at 19:46

you can accomplish in both these ways.

1.hadoop fs -get <HDFS file path> <Local system directory path>
2.hadoop fs -copyToLocal <HDFS file path> <Local system directory path>

Ex:

My files are located in /sourcedata/mydata.txt I want to copy file to Local file system in this path /user/ravi/mydata

hadoop fs -get /sourcedata/mydata.txt /user/ravi/mydata/

score 6 · Answer 5 · answered May 07 '19 at 17:16

6

If your source "file" is split up among multiple files (maybe as the result of map-reduce) that live in the same directory tree, you can copy that to a local file with:

hadoop fs -getmerge /hdfs/source/dir_root/ local/destination

answered May 07 '19 at 17:16

Eponymous

6,143
4
43
43

This should be accepted. This is what most people are looking for, not a split up file. – James O'Brien Aug 17 '19 at 19:18
This would be the best answer to be honest. Usually all HDFS files/tables are separated like 0000_0, 0001_0 in those directory. `-getmerge` will merge all those and put in into 1 files in local directory. Kudos to @Eponymous – didi Apr 19 '21 at 02:48

Zach · Answer 6 · 2018-03-06T22:32:11.503

3

This worked for me on my VM instance of Ubuntu.

hdfs dfs -copyToLocal [hadoop directory] [local directory]

edited Mar 06 '18 at 22:32

answered Mar 06 '18 at 15:00

Zach

51
7

score 1 · Answer 7 · answered Feb 25 '21 at 23:57

1

1.- Remember the name you gave to the file and instead of using hdfs dfs -put. Use 'get' instead. See below.

$hdfs dfs -get /output-fileFolderName-In-hdfs

answered Feb 25 '21 at 23:57

annye braca

11
1

score 0 · Answer 8 · answered Sep 03 '19 at 13:44

if you are using docker you have to do the following steps:

copy the file from hdfs to namenode (hadoop fs -get output/part-r-00000 /out_text). "/out_text" will be stored on the namenode.
copy the file from namenode to local disk by (docker cp namenode:/out_text output.txt)
output.txt will be there on your current working directory

score -3 · Answer 9 · edited Jan 31 '16 at 14:19

-3

bin/hadoop fs -put /localfs/destination/path /hdfs/source/path

edited Jan 31 '16 at 14:19

Brian Tompsett - 汤莱恩

5,753
72
57
129

answered Jan 29 '16 at 02:46

Hadoop maniac

29

1

hdfs dfs -put is a command to push files from to local FS to HDFS. hdfs dfs -get is the right option – b_rousseau Dec 10 '17 at 19:55

How to copy file from HDFS to the local file system

9 Answers9

Linked