0

I'm having some strange problems on Spark running with sparklyr.

I'm currently on an R production server, connecting to a my Spark Cluster in client mode via spark://<my server>:7077 and then pulling data from a MS SQL Server.

I was able to do this recently with no issues, but I recently was given a bigger cluster and am now having memory issues.

First I was getting inexplicable 'out of memory' errors during my processing. This happened a few times and then I started getting 'Out of memory unable to create new thread' errors. I checked the number of threads I was using compared to the max for my user on both the R production server and Spark server and I was no where near the max.

I restarted my master node and am now getting:

# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.

What the heck is going on??

Here are my specs:
- Spark Standalone running via root user.
- Spark version 2.2.1
- Sparklyr version 0.6.2
- Red Hat Linux

Zafar
  • 1,897
  • 15
  • 33

1 Answers1

2

I figured this out by chance. It turns out that when you are running operations on an external spark cluster on client mode it still runs Spark locally as well. I think that the local Spark did not have enough memory allocated and that was causing the error. My fix was simple:

Instead of allocating memory via:

spark_conf = spark_config()
spark_conf$`spark.driver.memory` <- "8G"
spark_conf$`spark.executor.memory` <- "12G"

I used:

spark_conf = spark_config()
spark_conf$`sparklyr.shell.driver-memory` <- "8G"
spark_conf$`sparklyr.shell.executor-memory` <- "12G"

The former will set resources on the cluster (spark context) directly. The latter sets it in the spark context as well as the rest of the sparklyr application.

Zafar
  • 1,897
  • 15
  • 33