Spark local testing reports OutOfMemoryError, how to fix?

Question

I followed this article to write some spark tests which running on local: http://mkuthan.github.io/blog/2015/03/01/spark-unit-testing/

The main code is like:

class SparkExampleSpec extends FlatSpec with BeforeAndAfter {

  private val master = "local[2]"
  private val appName = "example-spark"

  private var sc: SparkContext = _

  before {
    val conf = new SparkConf()
      .setMaster(master)
      .setAppName(appName)

    sc = new SparkContext(conf)
  }

  after {
    if (sc != null) {
      sc.stop()
    }
  }
  (...)

But I found when my tests increased, it will report OutOfMemoryError and the process is blocked.

How to fix it?

PS: Some of the tests generates some RDD, but doesn't collect them, like:

val rdd = sparkContext.makeRDD(someData)

// not calling `rdd.collect()`

Is it a possible problem?

See https://stackoverflow.com/questions/15280839/how-to-set-heap-size-for-sbt — Reactormonk, Aug 24 '15 at 07:20
Very sorry that at last we found the reason is we didn't set a proper permsize for scalatest-maven-plugin: http://stackoverflow.com/questions/32246485/how-to-set-the-permsize-for-scalatest-maven-plugin — Freewind, Aug 30 '15 at 01:32

score 1 · Accepted Answer · answered Aug 24 '15 at 07:37

I can't exactly answer, because i don't know your spark configuration

But, I think that memory settings are probably.

Check your configuration of driver(or executor) memory

$SPARK_PATH/conf/spark-default.conf
spark.driver.memory    
spark.executor.memory

You will be able to get help from this page.

http://spark.apache.org/docs/latest/configuration.html

Spark local testing reports OutOfMemoryError, how to fix?

1 Answers1