H2o Number of Executors not working

Question

I start the sparkling-shell with the following command.

./bin/sparkling-shell --num-executors 4 --executor-memory 4g --master yarn-client

I only ever get two executors. Is this an H2o problem, YARN problem, or Spark problem?

Mike

which version of Spark and Sparkling water are you using? Also could you exaplain "I only get two executors"? How are you checking it? — Mateusz Dymczyk, May 14 '16 at 19:28
Possible duplicate of [how to make executors run spark program by using --num-executors?](http://stackoverflow.com/questions/34352669/how-to-make-executors-run-spark-program-by-using-num-executors/34353466#34353466) — Glennie Helles Sindholt, May 15 '16 at 07:43
@GlennieHellesSindholt this actually might be more related to Spakling Water, unfortunately, from what my colleagues told me. We had similar issues and are working on it, would like OP to give us more info, though. — Mateusz Dymczyk, May 15 '16 at 11:53
I only get two executors because when the cluster does come up, the print out of the IPs that comes to the screen only shows two IPs. It looks like a completely healthy two node cluster. — uh_big_mike_boi, May 16 '16 at 23:54

score 1 · Answer 1 · answered Nov 08 '17 at 12:00

There can be multiple reasons for this behaviour.

YARN can give you only the amount of executors based on available resources ( memory, vcores ). If you ask for more then you have resources, it will give you max what it can.

It can be also case when you have dynamic allocation enabled. This means that that Spark will create new executors when they are needed.

In order to solve some technicalities in Sparkling Water we need to discover all available executors at the start of the application by creating artificial computation and trying to utilise the whole cluster. This might give you less number of executors as well.

I would suggest looking at https://github.com/h2oai/sparkling-water/blob/master/doc/tutorials/backends.rst where you can read more about the paragraph above and how it can be solved using so called external sparkling water backend.

You can also have a look here https://github.com/h2oai/sparkling-water/blob/master/doc/configuration/internal_backend_tuning.rst. This is Sparkling Water guide for tuning the configuration.

Kuba

score 0 · Accepted Answer · answered May 16 '16 at 23:58

I got over the problem by changing the following four values in cloudera manager

Setting                                  Value
yarn.scheduler.maximum-allocation-vcores 8  
yarn.nodemanager.resource.cpu-vcores     4 
yarn.nodemanager.resource.cpu-vcores     4 
yarn.scheduler.maximum-allocation-mb     16 GB

H2o Number of Executors not working

2 Answers2