Unable to read file from HDFS

Question

I am trying to read files from HDFS. I am using the following code:

val sam = sc.wholeTextFiles("hdfs://localhost:9000"+inputFolder,4)

I am getting the following error:

java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost, expected: hdfs://localhost:9000

I had referenced this question for adding the URI in my file path:
Cannot Read a file from HDFS using Spark

But I am still not able to read the file due t the above error. How can I resolve this?

I'm confused about the error, says you used `hdfs://localhost`, but `expected: hdfs://localhost:9000`, and that is what your code already is using. Are you sure HDFS is running and is on port 9000 and is on your localhost? — OneCricketeer, Nov 08 '16 at 19:57
Then it errors out file not found...which is ok since it is in HDFS — sarthak, Nov 08 '16 at 20:56
inputFolder is a path in HDFS file system...if I use it like `sc.wholeTextFiles(inputFolder,4)` then spark would search in the local file system rather than HDFS and so will not find this path. — sarthak, Nov 08 '16 at 21:11
It looks like you dont have specified HADOOP_CONF_DIR check [this](http://stackoverflow.com/a/27301040/2715438) — VladoDemcak, Nov 08 '16 at 21:22

score 1 · Accepted Answer · answered Nov 09 '16 at 04:49

1

Can you check adding winutils.exe in your system and setting a environment variable for the same . Spark needs winutils.exe to do hdfs operations.

answered Nov 09 '16 at 04:49

A srinivas

score 0 · Answer 2 · answered Nov 08 '16 at 20:31

0

Try using IP instead of localhost

answered Nov 08 '16 at 20:31

Anand

2 Answers2