11

I used this code

My error is:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

17/02/03 20:39:24 INFO SparkContext: Running Spark version 2.1.0

17/02/03 20:39:25 WARN NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable

17/02/03 20:39:25 WARN SparkConf: Detected deprecated memory fraction 
settings: [spark.storage.memoryFraction]. As of Spark 1.6, execution and  
storage memory management are unified. All memory fractions used in the old 
model are now deprecated and no longer read. If you wish to use the old 
memory management, you may explicitly enable `spark.memory.useLegacyMode` 
(not recommended).

17/02/03 20:39:25 ERROR SparkContext: Error initializing SparkContext.

org.apache.spark.SparkException: A master URL must be set in your 
configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:379)
at PCA$.main(PCA.scala:26)
at PCA.main(PCA.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

17/02/03 20:39:25 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:379)
at PCA$.main(PCA.scala:26)
at PCA.main(PCA.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
   sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at   
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

Process finished with exit code 1
Bastien Jansen
  • 8,756
  • 2
  • 35
  • 53
fakherzad
  • 111
  • 1
  • 1
  • 3
  • Possible duplicate of [Spark - Error "A master URL must be set in your configuration" when submitting an app](https://stackoverflow.com/questions/38008330/spark-error-a-master-url-must-be-set-in-your-configuration-when-submitting-a) – KARTHIKEYAN.A Sep 03 '17 at 19:26

4 Answers4

11

If you are running spark stand alone then

val conf = new SparkConf().setMaster("spark://master") //missing

and you can pass parameter while submit job 

spark-submit --master spark://master

If you are running spark local then 

val conf = new SparkConf().setMaster("local[2]") //missing

you can pass parameter while submit job 

spark-submit --master local

if you are running spark on yarn then 

spark-submit --master yarn
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Hutashan Chandrakar
  • 415
  • 4
  • 9
6

Error message is pretty clear, you have to provide the address of the Spark Master node, either via the SparkContext or via spark-submit:

val conf = 
  new SparkConf()
    .setAppName("ClusterScore")
    .setMaster("spark://172.1.1.1:7077") // <--- This is what's missing
    .set("spark.storage.memoryFraction", "1")

val sc = new SparkContext(conf)
Yuval Itzchakov
  • 146,575
  • 32
  • 257
  • 321
  • now I have another question about this code: how can i input my text . insted of "/data/kddcupdata/kddcup.trasfrom.nou2r" i want use my text file that was saved in "C://kddcup.data_10_percent_corrected.txt" . plz help me how can do it? – fakherzad Feb 03 '17 at 20:57
  • @fakherzad You can use `file:///kddcup.data_10_percent_corrected.txt` for reading files on your local machine. – OneCricketeer Feb 03 '17 at 22:42
  • thank s for your guide, but i have also the error :" Input path does not exist: file:/kddcup.data_10_percent_corrected.txt " . i dont know how can i solved it. plz help me – fakherzad Feb 04 '17 at 19:39
2
 SparkConf configuration = new SparkConf()
            .setAppName("Your Application Name")
            .setMaster("local");
 val sc = new SparkContext(conf);

It will work...

clemens
  • 16,716
  • 11
  • 50
  • 65
Shyam Gupta
  • 489
  • 4
  • 8
1

Most probably you are using Spark 2.x API in Java. Use code snippet like this to avoid this error. This is true when you are running Spark standalone on your computer using Shade plug-in which will import all the runtime libraries on your computer.

SparkSession spark = SparkSession.builder()
                .appName("Spark-Demo")//assign a name to the spark application
                .master("local[*]") //utilize all the available cores on local
                .getOrCreate();
Ashu
  • 614
  • 8
  • 17