1

For context, I'm trying to configure my Spark session to use fair scheduling. For that reason I have a file, fairscheduler.xml, at the 'root' of my resources folder. This file is correctly packaging into my build folder in the location I'd expect.

To further prove that's working, I have this line:

val mypath = getClass.getResource("/fairscheduler.xml")

That is working and returns exactly the path I'd expect: /<some path>/<my jar>/fairscheduler.xml

The following is throwing a FileNotFound exception:

sessionBuilder
    ...
    .config("spark.scheduler.mode", "FAIR")
    .config("spark.scheduler.allocation.file", mypath.toString) <- THIS LINE
    .config("spark.scheduler.pool", "fair_pool")
    ...
    .getOrCreate

Just for sanity, I have logged out mypath.toString.

Why is it that the spark session builder can't recognize a resource file that otherwise seems valid? Does it expect "spark configuration" files to exist somewhere specific? Or am I completely missing some dumb little thing here?

Adam
  • 877
  • 2
  • 10
  • 24
  • 1
    Did you check this question? https://stackoverflow.com/questions/941754/how-to-get-a-path-to-a-resource-in-a-java-jar-file – Emer Apr 16 '21 at 18:32
  • 1
    Thank you! I didn't know that resource files weren't necessarily in the file system. Writing a temp file seems to work :) Add that comment as an answer and I can upvote it. – Adam Apr 19 '21 at 16:02

1 Answers1

1

This is not a Apache Spark limitation but an expected behaviour of how resource files are handled in Java JAR files. Question How to get a path to a resource in a Java JAR file provides more details and a workaround of persisting the content in a temp file.

Emer
  • 3,734
  • 2
  • 33
  • 47