0

When I try to run my code (Scala, Spark) in debug mode, it is way slower than when I "run" it. Would you know why?

I use Scala 2.10, Spark 2.2 and IntelliJ 2016.3.5

belka
  • 1,480
  • 1
  • 18
  • 31
  • Might be related to few things.. For starter, have you verified how many core are used ? You can increase it with programatically with 'spark.master' on your sparkContext or system properties. Also, spark debug logger output is especially verbose and that might slow down everything. – Michel Lemay Aug 23 '17 at 14:28
  • See related question: https://stackoverflow.com/questions/2195720/why-does-java-code-slow-down-in-debugger. That question refers to Java, but since Scala compiles into the same bytecode and runs on JVM, the same answer applies. Not posting this as an answer since I'm not sure there are no additional Scala-specific causes at play here... – Tzach Zohar Aug 23 '17 at 17:53
  • @TzachZohar Yes, I wonder if there is not anything related to functionnal programming, very specifically – belka Aug 24 '17 at 07:54

0 Answers0