How to resolve spark processnext() error on spark-submit with Pyspark

Asked Jun 11 '20 at 17:24

Active Jun 11 '20 at 17:24

Viewed 1,396 times

I'm running a spark-submit Pyspark job and have been running into this error a lot recently. Does anyone if this can be resolved with the settings?

ERROR codegen.CodeGenerator: failed to compile: org.codehaus.janino.InternalCompilerException: Compiling "GeneratedClass": Code of method "processNext
()V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1" grows beyond 64 KB org.codehaus.janino.InternalCompilerException: Compiling "GeneratedClass": Code of method "processNext()V" of class "org.apache.spark.sql.catalyst.expressions.Generated
Class$GeneratedIteratorForCodegenStage1" grows beyond 64 KB

asked Jun 11 '20 at 17:24

algorythms

1,547
1
15
28

1

does [this question](https://stackoverflow.com/q/50891509/2129801) help you? – werner Jun 11 '20 at 19:23

How to resolve spark processnext() error on spark-submit with Pyspark

0 Answers0