Using Sparklyr, I'm trying to do a spark_connect(master="yarn")
, but I'm receiving this error:
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps
...
Error in spark_connect_gateway(gatewayAddress, gatewayPort,sessionId, :
Gateway in localhost:8880 did not respond
I have Spark version 2.4.8 and Scala version 2.12.10. I'm running on a Google DataProc cluster using these version specs. I'm using Sparklyr version 1.3.1.