Spark on NUMA systems

Question

I'm considering Apache Spark for data analysis. In the past I've experienced Java/Scala slowdowns on 4-socket servers due to NUMA architecture and objects being local to a single node. The solution was to start a separate pinned JVM for each NUMA node and have them talk to each other using Akka.

How will NUMA be handelled in Spark to avoid similar situations?

score 2 · Accepted Answer · answered Mar 01 '15 at 12:24

2

If you start Spark with --executor-cores=32 (assuming 8 virtual cores per socket) you will have the same issues. But you can start 4 workers per machine, each with --executor-cores=8 instead. Then you could pin these executors to the nodes.

This setup will incur more communication overhead, but will likely be a good trade-off. Spark tries to minimize communication between executors, since they on different machines in the typical case.

answered Mar 01 '15 at 12:24

Daniel Darabos

26,991
10
102
114

I have tried running more workers per machine and it works fine. I haven't experimented with NUMA performance though, so that part is just my guess. Hope it help! – Daniel Darabos Mar 01 '15 at 12:25

Spark on NUMA systems

1 Answers1