Infinite recursion (StackOverflowError) in Amazon EMR cluster

Asked Sep 10 '18 at 17:00

Active Sep 10 '18 at 17:00

Viewed 376 times

We are running a Spark App on an Amazon EMR cluster. Our code takes in JSON data, converts it to dataframes, carries out Spark SQL transformations on them and then writes the resulting dataframes to csv files (at least, it tries to). However, when executing the code on an EMR cluster, we get the following error:

ERROR AsyncEventQueue: Listener EventLoggingListener threw an exception
com.fasterxml.jackson.databind.JsonMappingException: Infinite recursion 
(StackOverflowError) (through reference chain: 
scala.collection.convert.IterableWrapper[0]->org.apache.spark.sql.execution.SparkPlanInfo["children"]->scala.collection.convert.IterableWrapper[0]->

...etc

Can anyone help us? Thank you very much in advance.

asked Sep 10 '18 at 17:00

JOSE MANUEL RAMIREZ LEON

Have you successfully run the code in other places? – John Humphreys Sep 10 '18 at 17:03
Actually, yes, locally on our PCs – JOSE MANUEL RAMIREZ LEON Sep 10 '18 at 17:05
It’s stackoverflow error due to recursion. I doubt it worked in another environment.could you please change your code to use tail recursion so that it won’t create a different stack frame for each call – Chandan Ray Sep 10 '18 at 17:07
Thank you very much for your insight, we will try to use tail recursion :) – JOSE MANUEL RAMIREZ LEON Sep 10 '18 at 17:17
Code snippets are needed. To address this kind of problems – Ram Ghadiyaram Sep 12 '18 at 01:22

Infinite recursion (StackOverflowError) in Amazon EMR cluster

0 Answers0