1

I'm wondering how does a YARN app (let's say MapReduce job) estimate needed resources (CPU, RAM) for a single mapper/reducer.

facha
  • 11,862
  • 14
  • 59
  • 82

1 Answers1

0

The question is too broad but I'll try to give a direction for investigation. When Yarn application is executed, it requests some number of resources from Resource Manager. Resource management in Yarn is implemented by means of schedulers. Yarn supports two schedulers:

Schedulers define rules that are used for estimating "slots" for an application. For some schedulers "slots" are defined only by memory required for the application (Capacity Scheduler with DefaultResourseCalculator). Others take number of CPUs into account as well.

Dmitry Y.
  • 185
  • 8
  • My question is about the memory requirements for a single mapper/reducer. Let's say a node where a mapper is about to be launched has unlimited cores/memory available and there are no other containers running. How much memory will a single mapper ask for its jvm? – facha Apr 19 '16 at 08:10
  • Memory can be configured. You can configure container size for map tasks and reduce tasks as well as specify upper limits for them. – Dmitry Y. Apr 19 '16 at 08:19
  • You configure the upper limit. However if your upper limit is infinity (let's say 2TB of RAM) will a single mapper ask for all that memory? If not, how much memory will be asked for? – facha Apr 19 '16 at 08:21
  • In mapred-site.xml `mapreduce.map.memory.mb 4096 mapreduce.reduce.memory.mb 8192` `mapreduce.map.java.opts -Xmx3072m mapreduce.reduce.java.opts -Xmx6144m` See [this page](http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/) for more details: – Dmitry Y. Apr 19 '16 at 08:24
  • Also, check [this discussion on SO](http://stackoverflow.com/questions/24070557/what-is-the-relation-between-mapreduce-map-memory-mb-and-mapred-map-child-jav) – Dmitry Y. Apr 19 '16 at 08:27
  • So, underlining, we can specify upper limits for containers and upper limit of the physical RAM that Map and Reduce tasks will use. – Dmitry Y. Apr 19 '16 at 08:30
  • If the values above are not defined explicitly, default values will be used. – Dmitry Y. Apr 19 '16 at 08:31