Error when trying to load a file with sc.textFile in Windows

Question

I am new to hadoop and I am trying to upload a local file using sc.textFile command

val data = sc.textFile("file:///D:\\test.txt")

After this I try to do some operation on this data then I get the error

java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: D:test.txt
        at org.apache.hadoop.fs.Path.initialize(Path.java:206)
        at org.apache.hadoop.fs.Path.<init>(Path.java:172)
        at org.apache.hadoop.fs.Path.<init>(Path.java:94)
        at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
        at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1698)
        at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:229)
        at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:200)
        at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270)
        at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:179)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
        at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
        at org.apache.spark.rdd.FlatMappedRDD.getPartitions(FlatMappedRDD.scala:30)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
        at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
        at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:65)
        at org.apache.spark.rdd.PairRDDFunctions.reduceByKey(PairRDDFunctions.scala:279)
        at $iwC$$iwC$$iwC$$iwC.<init>(<console>:16)
        at $iwC$$iwC$$iwC.<init>(<console>:21)
        at $iwC$$iwC.<init>(<console>:23)
        at $iwC.<init>(<console>:25)
        at <init>(<console>:27)
        at .<init>(<console>:31)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:814)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:859)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:771)
        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:616)
        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:624)
        at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:629)
        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:954)
        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:902)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:997)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: D:test.txt
        at java.net.URI.checkPath(URI.java:1804)
        at java.net.URI.<init>(URI.java:752)
        at org.apache.hadoop.fs.Path.initialize(Path.java:203)
        ... 69 more

I have tried a few variants in the command sc.textFile. Such as

val data = sc.textFile("file:///D:/test.txt")

then I get the error

java.lang.IllegalArgumentException: java.net.URISyntaxException: Expected scheme-specific part at index 2: D

I have been trying for a long time but I am not able to get this basic thing correct. Can someone please help !! Thanks in advance !!

Try just "D:\\test.txt". take a look at this - http://stackoverflow.com/questions/30520176/how-to-access-local-files-in-spark-on-windows — Pranav Shukla, Jun 08 '16 at 08:05
@AvihooMamka This gives the error java.lang.IllegalArgumentException: Wrong FS: file://D:/test.txt, expected: file:/// — varunkr, Jun 08 '16 at 08:08
@PranavShukla This will just generate just a string variable, ultimately to use other commands in further computation I will have to use sc.textFile() — varunkr, Jun 08 '16 at 08:23
@varunkr I meant the argument to `sc.textFile()` i.e. `sc.textFile("D:\\test.txt")` — Pranav Shukla, Jun 08 '16 at 08:25
@PranavShukla That too gives the relative path error as I mentioned in the question :( — varunkr, Jun 08 '16 at 08:34
@AvihooMamka Tried that now, that too gives relative path exception :( — varunkr, Jun 08 '16 at 09:01
I tried above commands, all work fine with spark 1.6.0 on windows. — Nishu Tayal, Jun 08 '16 at 10:14
@NishuTayal The commands don't give an error but when I do some operation on the variable data, such as finding the count of words etc then I get the error. — varunkr, Jun 08 '16 at 11:50
@varunkr :did you try to print the data using data.collect() first? — Nishu Tayal, Jun 08 '16 at 11:54
@NishuTayal I just trying printing the data via data.collect() and gives me the same error, relative path in absolute uri. I am using the command val data = sc.textFile("file:///D:\\test.txt") :( — varunkr, Jun 09 '16 at 08:27
I think it is a duplicate of http://stackoverflow.com/q/38940312/1305344 — Jacek Laskowski, Aug 14 '16 at 19:40

score 1 · Answer 1 · answered Mar 30 '17 at 23:09

Adding spark.sql.warehouse.dir fixed the problem for me

System.setProperty("spark.sql.warehouse.dir", "file:///C:/spark-warehouse");

Here is the code snippet

System.setProperty("hadoop.home.dir", "c:/winutil/");
System.setProperty("spark.sql.warehouse.dir", "file:///C:/spark-warehouse");
val conf = new SparkConf().setAppName("test").setMaster("local[*]")
val sc = new SparkContext(conf)
val lines = sc.textFile("C:/user.txt")

Error when trying to load a file with sc.textFile in Windows

1 Answers1