2

I have:

val sparkBuilder: SparkSession.Builder = SparkSession
  .builder
  .appName("CreateModelDataPreparation")
  .config("spark.master", "local")
implicit val spark: SparkSession = sparkBuilder.getOrCreate()

However, when I run my program I still get:

org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:379)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)

The SparkSession is set in the Main method as suggested in other posts. Those did not seem to solve the problem.

This differs from the suggested duplicate as I have tried both:

  def main(argv: Array[String]): Unit = {
    import DeweyConfigs.implicits.da3wConfig

    val commandlineArgs: DeweyReaderArgs = processCommandLineArgs(argv)

    val sparkBuilder: SparkSession.Builder = SparkSession
      .builder
      .appName("CreateModelDataPreparation")
      .master("local")
    implicit val spark: SparkSession = sparkBuilder.config("spark.master", "local").getOrCreate()
    import spark.implicits._
    ...

and

  def main(argv: Array[String]): Unit = {
    import DeweyConfigs.implicits.da3wConfig

    val commandlineArgs: DeweyReaderArgs = processCommandLineArgs(argv)

    val sparkBuilder: SparkSession.Builder = SparkSession
      .builder
      .appName("CreateModelDataPreparation")
      .config("master", "local")
    implicit val spark: SparkSession = sparkBuilder.config("spark.master", "local").getOrCreate()
    import spark.implicits._
    ...
Matthew Hoggan
  • 7,402
  • 16
  • 75
  • 140
  • This already answered post might help you out [A master URL must be set in your configuration](https://stackoverflow.com/questions/38008330/spark-error-a-master-url-must-be-set-in-your-configuration-when-submitting-a) –  Aug 30 '18 at 13:17
  • You're right, marked it as a duplicate. – John Humphreys Aug 30 '18 at 13:21
  • @JohnHumphreys-w00te The suggestions in that post did not work. Is there a way to reopen that post or append my comments to it so as to get the communities attention? – Matthew Hoggan Aug 30 '18 at 13:33
  • Sure, I'll vote to re-open this now that you've clarified all that. Some other people will have to come agree in a bit; its similar to the closing process. – John Humphreys Aug 30 '18 at 13:55
  • Can you please provide a MVCE then if this is not a dupe ? With the current shared information, it's most likely unsalvageable. I'll gladly reopen it if it's not a dupe. – eliasah Aug 30 '18 at 14:36

1 Answers1

5

Try adding .master("local") on the builder as opposed to the config parameter you provided.

I would have thought they did the same thing, but I'm pretty sure the latter works.

John Humphreys
  • 37,047
  • 37
  • 155
  • 255
  • Marking as duplicate, this suggestion is actually (basically) already in :https://stackoverflow.com/questions/38008330/spark-error-a-master-url-must-be-set-in-your-configuration-when-submitting-a as pointed otu by Sundeep. – John Humphreys Aug 30 '18 at 13:21