Apache spark master URL error running in idea intellij

Question

i am using intellij idea for spark application in scala. i am not sure why i am getting below error.

Code:

package batch

import java.lang.management.ManagementFactory

import org.apache.spark.{SparkConf, SparkContext}

object BatchJob {
  def main(args: Array[String]): Unit = {


    val conf = new SparkConf()
      .setAppName("Lambda with Spark")

    val sc = new SparkContext(conf)
    val sourceFile = "file:///E:/Boxes/data.tsv"
    val input = sc.textFile(sourceFile)

    input.foreach(println)
  }

}

Error:

**Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration**

@TerryDactyl: its throwing error as : Error:(22, 31) not found: value conf val sc = new SparkContext(conf) — Rohini Mathur, Oct 09 '18 at 06:17
Does this answer your question? [Spark - Error "A master URL must be set in your configuration" when submitting an app](https://stackoverflow.com/questions/38008330/spark-error-a-master-url-must-be-set-in-your-configuration-when-submitting-a) — Carlos Verdes, Jun 15 '20 at 17:16

score 1 · Answer 1 · answered Jun 15 '20 at 11:30

Previous answers force the driver to run in local with a number of cores. This is fine to do a quick test on local but is not good if you want to execute this driver with Spark submit command.

The trick that worked for me is to create a default conf and check if master is defined. If it's not then I assume is a local test and I force master = local.

val SPARK_MASTER = "spark.master"
val DEFAULT_MASTER = "local[*]"

// get default conf
val defaultConf = new SparkConf()

// if master is not defined then set default to local
if(!defaultConf.contains(SPARK_MASTER)) defaultConf.setMaster(DEFAULT_MASTER)

val sparkSession = SparkSession.builder().config(defaultConf).getOrCreate()

Soheil Pourbafrani · Accepted Answer · 2018-10-09T08:26:17.447

0

If you're running Spark on Intellij (in local mode), You should set master for the Spark config object, too:

val conf = new SparkConf().setAppName("Lambda with Spark").setMaster("local[*]")

edited Oct 09 '18 at 08:26

answered Oct 09 '18 at 08:14

Soheil Pourbafrani

3,249
3
32
69

This will override master configuration if this is called from spark-submit, check my answer which I think is better – Carlos Verdes Jun 15 '20 at 11:24
@CarlosVerdes sure, but the question is about running Spark on Intellij, not using spark-submit command! – Soheil Pourbafrani Jun 15 '20 at 14:06
I run my code inside intellij, check my answer I tested today and is working :) – Carlos Verdes Jun 15 '20 at 17:09

score 0 · Answer 3 · answered Oct 09 '18 at 08:19

The problem is exactly what the error says ,u have to set a master url for spark to run.If you are running inside ide or locally you should make SparkConf object like this:

val conf = new SparkConf()
  .setAppName("Lambda with Spark").setMaster(local[*])

for running on clusters you can use 'yarn' as the master.

Apache spark master URL error running in idea intellij

3 Answers3