9

We are trying to run our spark cluster on yarn. We are having some performance issues especially when compared to the standalone mode.

We have a cluster of 5 nodes with each having 16GB RAM and 8 cores each. We have configured the minimum container size as 3GB and maximum as 14GB in yarn-site.xml. When submitting the job to yarn-cluster we supply number of executor = 10, memory of executor =14 GB. According to my understanding our job should be allocated 4 container of 14GB. But the spark UI shows only 3 container of 7.2GB each.

We are unable to ensure the container number and resources allocated to it. This causes detrimental performance when compared to the standalone mode.

Can you drop any pointer on how to optimize yarn performance?

This is the command I use for submitting the job: 

$SPARK_HOME/bin/spark-submit --class "MyApp" --master yarn-cluster --num-executors 10 --executor-memory 14g  target/scala-2.10/my-application_2.10-1.0.jar

Following the discussion I changed my yarn-site.xml file and also the spark-submit command.

Here is the new yarn-site.xml code :

<property>
<name>yarn.resourcemanager.hostname</name>
<value>hm41</value>
</property>

<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>14336</value>
</property>

<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2560</value>
</property>

<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>13312</value>
</property>

And the new command for spark submit is

$SPARK_HOME/bin/spark-submit --class "MyApp" --master yarn-cluster --num-executors 4 --executor-memory  10g --executor-cores 6   target/scala-2.10/my-application_2.10-1.0.jar

With this I am able to get 6 cores on each machine but the memory usage of each node is still around 5G. I have attached the screen shot of SPARKUI and htop. enter image description here Spark UI Screenshot![][1]

3 Answers3

3

The memory (7.2GB) you see in the SparkUI is the spark.storage.memoryFraction, which by default is 0.6. As for your missing executors, you should look in the YARN resource manager logs.

Sietse
  • 201
  • 1
  • 4
  • In fact, not really 0.6. It is 0.6 of the "safe memory", which is 0.9 of the whole heap, so by default it is 0.54 of the JVM heap – 0x0FFF Feb 06 '15 at 15:18
  • Sure, and while we're at it, 14GB is not really 14GB in YARN, but 14GB + memoryOverhead. But that's not what he's asking right? – Sietse Feb 06 '15 at 15:42
1
  1. Withing yarn-site.xml check that yarn.nodemanager.resource.memory-mb is set the right way. In my understanding of your cluster it should be set to 14GB. This setting is responsible for giving the YARN know how much memory it can use on this specific node
  2. If you have this set right and you have 5 servers running YARN NodeManager, then your job submission command is wrong. First, --num-executors is the number of YARN containers would be started for executing on the cluster. You specify 10 containers with 14GB RAM each, but you don't have this many resources on your cluster! Second, you specify --master yarn-cluster, which means that Spark Driver would run inside of the YARN Application Master that would require a separate container.
  3. In my opinion it shows 3 containers because out of 5 nodes in the cluster you have only 4 of them running YARN NodeManager + you request to allocate 14GB for each of the containers, so YARN first starts Application Master and then polls the NM for available resources and see that it can start only 3 containers. Regarding heap size you see, after starting the Spark find its JVM containers and see the parameters of their start - you should have many -Xmx flags in a single line - one correct and one wrong, you should find its origin in config files (Hadoop or Spark)
  4. Before submitting an application to the cluster, start the spark-shell with the same settings (replace yarn-cluster with yarn-client) and check how it is started, check WebUI and JVMs started
0x0FFF
  • 4,948
  • 3
  • 20
  • 26
  • My {yarn.nodemanager.resource.memory-mb} is 15GB, as we leave 1GB for the OS processes and allow the nodemangaer to distribute the other 15GB. I modified my submit call to this. --master yarn-cluster --num-executors 5 --executor-memory 13g –  Feb 06 '15 at 10:50
  • I suspect that together with NM itself you also run DataNode, so 15GB in my opinion is too much, I wouldn't go beyond 14GB – 0x0FFF Feb 06 '15 at 10:56
  • Can I ascertain during/after container creation what is the amount of RAM a container was assigned? I tried going through the logs of resource manager but was unable to pinpoint the exact entries for it. Our cluster is not a production or a busy one so its ok if we can ensure that spark gets all the RAM it can. @sietse Au Does that means that spark container get the required memory but report only that fraction? because in our standalone implementation the whole memory is reported. –  Feb 06 '15 at 16:13
  • Login to the YARN NM machine, run `ps -ef | grep java | more` and find the Spark executor container, look at `-Xmx` parameter – 0x0FFF Feb 06 '15 at 16:23
0

Just because YARN "thinks" it has 70GB (14GBx5), doesn't mean at run time there is 70GB available on the cluster. You could be running other Hadoop components (hive, HBase, flume, solr, or your own app, etc.) which consume memory. So the run-time decision YARN makes is based on what's currently available -- and it had only 52GB (3x14GB) available to you. By the way, the GB numbers are approximate because it is really computed as 1024MB per GB...so you will see decimals.

Use nmon or top to see what else is using memory on each node.

jfchen
  • 21
  • 3