0

I set my maximum allocation vcores to 1 in yarn-site.xml, and the number of physical cores to 2. All nodes are the same. However, sometims, YARN allocates 3 containers to a node. It reports that -1 vcores are available on that node. What ends up happening is that nodemanager shuts down on that node, and the job fails. How do you prevent YARN from over allocating vcores? There are other nodes available, with 0 vcores used. I already specified the minimum and maximum amount of vcores that can be allocated and the number of physical vcores per node. When I look at the YARN reports, everything looks right.

spearmunkie
  • 21
  • 1
  • 7

1 Answers1

1

This answer is what you are looking for. Unfortunately, it seems like the default YARN behaviour is to only consider memory when doing these allocations. Luckily this can be changed, see the linked answer.

I feel cheated by all the documentation on YARN as nothing I've found mentions this change is required, yet there are plenty of blogs/posts on how to set the vcore value.

Community
  • 1
  • 1
jonson
  • 510
  • 5
  • 9