Apache Spark Submit Error

Question

I am new to Apache Spark and testing my first program.
It is a 2-3 lines program just for testing purposes.

I am using Eclipse and compiled the java file with Maven.
I am trying to run the spark-submit but getting this error.

I do not think it is from the file name or the path.
Could it be from another issue?

...spark-2.1.0-bin-hadoop2.7\bin>spark-submit --class "Main" --master local[4] "C:\Users\...\target\SparkTest-0.0.1-SNAPSHOT.jar"

The filename, directory name, or volume label syntax is incorrect.

This is the main class

import java.util.Arrays;

import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;

import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;

import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.PairFunction;



public class SparkMain {

    public static void main(String[] args) {

        SparkConf conf = new SparkConf().setMaster("local").setAppName("My App");
        JavaSparkContext sc = new JavaSparkContext(conf);
        System.out.println("HELLO");


        JavaRDD<String> lines = sc.textFile("C:/spark/spark-2.1.0-bin-hadoop2.7/README.md");

        System.out.println(lines.count());


    }

}

You don't need quotes on `Main` (because Java classes can't have spaces or start with dashes), but this looks fine, so what is the actual error message? — OneCricketeer, Jan 23 '17 at 23:44
@cricket_007. I press Enter it gives me : The filename, directory name, or volume label syntax is incorrect. I tried with escaping the backslashes, same error. the directories are correct. Maybe it is a windows issue ? — Missak Boyajian, Jan 24 '17 at 19:09
1) Can you show the Main class? It could be your Spark code throwing the error 2) Try to see if the `SparkPi` example runs to test Spark works. 3) `cd` to the `target` directory and use the full path to `spark-submit` so that you don't need to give the full path to the JAR — OneCricketeer, Jan 24 '17 at 20:37
@cricket_007 I edited the question. I tried with // still same error. — Missak Boyajian, Jan 24 '17 at 20:58
Possible duplicate of [How to access local files in Spark on Windows?](http://stackoverflow.com/questions/30520176/how-to-access-local-files-in-spark-on-windows) — OneCricketeer, Jan 24 '17 at 22:33

OneCricketeer · Answer 1 · 2017-01-24T22:42:37.770

You are passing a String to a Java program and on a Windows machine.

Windows uses backslashes and need to be escaped.

I'm on a Mac, so this is hard to test, but you could try something like this.

import java.nio.file.Paths;

...

String fileName = Paths.get("C:", "spark", "spark-2.1.0-bin-hadoop2.7", "README.md").toString()
JavaRDD<String> rdd = sc.textFile(fileName);
System.out.println(rdd.count());

If you want to be cross-platform, then perhaps this

String rootDir = Paths.get(System.getProperty("user.home")).getRoot().toString();
String fileName = Paths.get(rootDir, "spark", ...);
...

Refer: Java Essentials | Path Operations

Apache Spark Submit Error

1 Answers1