1

I am using spark in local mode:

trait LocalStandAloneSpark {
  val spark: SparkSession =
    SparkSession
      .builder
      .appName("local-mode-spark")
      .master("local[*]")
      .config("spark.executor.cores", 2)
      .config("spark.network.timeout", "10000001") // to avoid shutdown during debug, avoid otherwise
      .config("spark.executor.heartbeatInterval", "10000000") // to avoid shutdown during debug, avoid otherwise
      .getOrCreate()
}

And I'm probably not aware how to properly accomplish proper termination with error messages about interrupted operations. If I do not include spark.stop() at the end of my flow, I get the error from here, and when I do include spart.stop() there, I get this error:

[org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner] WARN org.apache.hadoop.fs.FileSystem - exception in the cleaner thread but it will continue to run
java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
    at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3212)
    at java.lang.Thread.run(Thread.java:748)

I suspect that maybe the use of java's FileUtils for writing files as part of my code might be related, but am not sure how to go about it (I'm not explicitly using any asynchronous FileUtils functions).

What should I do in order to avoid that kind of error?

As much as that's relevant, I'm running my main via sbt.

matanster
  • 15,072
  • 19
  • 88
  • 167
  • Have you tried to run it in forked JVM? (in sbt add `fork in run := true`). – blackbishop Dec 28 '19 at 12:31
  • You are right! I thought I was using that setting but I wasn't. Care to discuss why would that matter? – matanster Dec 28 '19 at 13:33
  • 2
    This is explained in this [answer](https://stackoverflow.com/questions/44298847/why-do-we-need-to-add-fork-in-run-true-when-running-spark-sbt-application#answer-47172931). – blackbishop Dec 28 '19 at 14:00

0 Answers0