How to read the file from hdfs

Question

I have a text file name mr.txt in the hadoop file sytem under /project1 directory. I need to write the python code to read the first line of the text file without downloading mr.txt file into local. But I have trouble to open the mr.txt file from hdfs. I had tried:

open('hdfs:///project1/mr.txt','r')

"Without downloading `mr.txt`"... Well, you have to do that to actually gather the file on some node. — OneCricketeer, Feb 24 '17 at 21:24

score 4 · Accepted Answer · answered Feb 24 '17 at 21:26

4

Get PySpark installed.

text = sc.textFile('hdfs:///project1/mr.txt')
first_line = text.first()

answered Feb 24 '17 at 21:26

OneCricketeer

179,855
19
132
245

Thank you @cricket_007 . It worked – gd1 Feb 24 '17 at 21:48

score 0 · Answer 2 · edited Feb 24 '17 at 21:23

0

Without knowing in more detail what your software is or where it is run...

You can use a NFS server so you can mount the HDFS volume and access to it locally. If this option does not suit your needs, you should use Hadoop Streaming. Finally if you are writting a Spark job, you can access the HDFS as if it were your local FS.

edited Feb 24 '17 at 21:23

OneCricketeer

179,855
19
132
245

answered Feb 24 '17 at 20:58

Navarro

1,284
2
17
40

How to read the file from hdfs

2 Answers2

Linked