11

I'm running a spark job written in Scala 2.10.4 and running on Spark 1.4.0 cluster (based on HDFS and managed with YARN) and using Jackson modules version 2.6.1 on Maven repository

When running the code locally from my IDE (IntelliJ IDEA v14) everything works on the on-memory cluster, but when running the job on my remote cluster (EMR cluster on AWS VPC) i'm getting the following exception:

java.lang.AbstractMethodError: com.company.scala.framework.utils.JsonParser$$anon$1.com$fasterxml$jackson$module$scala$experimental$ScalaObjectMapper$_setter_$com$fasterxml$jackson$module$scala$experimental$ScalaObjectMapper$$typeCache_$eq(Lorg/spark-project/guava/cache/LoadingCache;)V
    at com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper$class.$init$(ScalaObjectMapper.scala:50)
    at com.company.scala.framework.utils.JsonParser$$anon$1.<init>(JsonParser.scala:14)
    at com.company.scala.framework.utils.JsonParser$.<init>(JsonParser.scala:14)
    at com.company.scala.framework.utils.JsonParser$.<clinit>(JsonParser.scala)
    at com.company.migration.Migration$.printAllKeys(Migration.scala:21)
    at com.company.migration.Main$$anonfun$main$1.apply(Main.scala:22)
    at com.company.migration.Main$$anonfun$main$1.apply(Main.scala:22)
    at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:199)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:56)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:70)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

I tried to look over the web for the exception and with no luck. I also tried look for a similar question here and found just one thread with no acceptable answer and none of the answers helped me there.

Hope to find help here,

Thanks.

Avihoo Mamka
  • 4,656
  • 3
  • 31
  • 44
  • I ran into the same issue. How did you solve this? – Aliza Sep 06 '15 at 05:10
  • Sadly, I didn't. I stopped using the ScalaObjectMapper and started working with the regular ObjectMapper. So far it achieves my goals. – Avihoo Mamka Sep 06 '15 at 06:12
  • What is the solution to work around? Thanks. I have this issue too. – giaosudau Sep 06 '15 at 14:16
  • 5
    I have this: `val jacksonMapper = new ObjectMapper() jacksonMapper.registerModule(DefaultScalaModule)` And it works fine for the time being. – Avihoo Mamka Sep 06 '15 at 14:33
  • 1
    Confirmed this works. The only difference in code is to use classOf[...] to specify type for readValue as the 2nd parameter. No big deal – piggybox Oct 28 '15 at 07:42

1 Answers1

12

I'm answering the question for further views by other users.

I stopped using the ScalaObjectMapper and started working with the regular ObjectMapper. 

val jacksonMapper = new ObjectMapper() 
jacksonMapper.registerModule(DefaultScalaModule)

And it works fine for the time being. Attaching piggybox's comment to be helpful comment:

The only difference in code is to use classOf[...] to specify type for readValue as the 2nd parameter.

Avihoo Mamka
  • 4,656
  • 3
  • 31
  • 44