-2

I am trying to read a text file from local path using spark. But it's throwing an exception.Error image

The code I used to read file is this:

val assetFile = sc.textFile(assetFilePath)

assestFilePath is a variable whichrepresent the path to somehere in my local machine.

val adFile = sc.textFile(adFilePath)

adFilePath is a variable whichrepresent the path to somehere in my local machine.

Akhil Choudhari
  • 3
  • 5
  • I highly recommend you to post the error stack and remove the link with the image. – dbustosp Mar 04 '17 at 02:13
  • Did you check this link: [http://stackoverflow.com/questions/27299923/how-to-load-local-file-in-sc-textfile-instead-of-hdfs](http://stackoverflow.com/questions/27299923/how-to-load-local-file-in-sc-textfile-instead-of-hdfs) – dbustosp Mar 04 '17 at 02:16

1 Answers1

0

sc.textFile will by default read from HDFS not from local file system, but Spark supports multiple file system apart from HDFS like LocalFileSyetem, Amazon S3, Azure, Swift FS.

So in order to read from the local file system you need to specify in the file path as protocol. for example :

sc.textFile("file:///tmp/myfile.txt")

This will read a file named myfile.txt from tmp directory present in local file system where spark driver code is running.

RBanerjee
  • 957
  • 1
  • 9
  • 18