How to access local files in Spark on Windows?

Question

I am using Spark on Windows. I know in *nix, accessing local file code likes this:

val textFile = sc.textFile("file:///usr/local/spark/README.md")

But how can I access a local file on Windows? I have tried following methods:

val logFile = "C:\spark-1.3.1-bin-hadoop2.4\README.md"
val logFile = "file\\C:\spark-1.3.1-bin-hadoop2.4\README.md"

But all can't work.

score 13 · Accepted Answer · edited Sep 22 '16 at 14:55

13

Unfortunately in windows you have to escape "\".

Try:

"C:\\spark-1.3.1-bin-hadoop2.4\\README.md"

edited Sep 22 '16 at 14:55

OneCricketeer

answered May 29 '15 at 05:04

ayan guha

Cool! You are right! It should be "`val logFile = "C:\\spark-1.3.1-bin-hadoop2.4\\README.md"`". – Nan Xiao May 29 '15 at 10:20
I am trying to configure below path as spark streaming source. Not working andy clue? new File(".").getCanonicalPath() + "\\data" – Joy George Kunjikkuru Jul 16 '18 at 20:52

score 2 · Answer 2 · answered Aug 27 '18 at 15:21

2

It should work with below code. Else try checking for spelling and correct path.

val path = "C:\\spark_home\\spark-2.2.0-bin-hadoop2.7\\README.md"
val read = sc.textFile(path)

answered Aug 27 '18 at 15:21

Arun Goudar

score 2 · Answer 3 · answered Dec 28 '18 at 03:28

2

In windows you have to specify as follows:

"file///C:/spark-1.3.1-bin-hadoop2.4/README.md"

answered Dec 28 '18 at 03:28

Sushil Ks

I write as you suggested and I get AnalysisException: 'Path does not exist: file:/server444/files/randomtest.csv – Amir Aug 22 '19 at 12:48
1

It should be `"file:///c:/xx/xx.txt"`. You are missing `:`. – Spaceship222 Jan 20 '21 at 11:28

score 1 · Answer 4 · answered Nov 20 '19 at 16:22

1

For CSV and txt file dont specify the format,

val file = "C:\\Users\\testUser\\IdeaProjects\\SparkDataQualityReporting\\SampleData"
val fileRDD = sparkSession.sparkContext.textFile(file)

answered Nov 20 '19 at 16:22

Jacob Joy

score 0 · Answer 5 · answered Nov 29 '22 at 08:05

0

In windows, we can specify as:

spark.read.format("json").load("file:///C:/workspace/sample_data.json")

answered Nov 29 '22 at 08:05

newbie

5 Answers5