5

Would someone know and explain to me please the difference between these settings of Tez ?

hive.tez.container.size and tez.task.resource.memory.mb

thanks.

Ulky Igor
  • 322
  • 5
  • 16

1 Answers1

3

hive.tez.container.size

This property specifies tez container size. Usually value of this property should be the same as or a small multiple (1 or 2 times that) of YARN container size yarn.scheduler.minimum-allocation-mb and should not exceed value of yarn.scheduler.maximum-allocation-mb.

As a general rule don't put value higher than memory per processor as you want 1 processor per container and you want to spun up multiple containers.

You can find very detailed answer and a great architecture diagram on Hortonworks community answer here

tez.task.resource.memory.mb

Amount of memory used by launched task in TEZ container. Usually this value is being set in DAG but if it's not then by default it will get set to 1024 MB.

All task in the DAG will have same size, it is helpful for container reuse and good for performance too.

Jainik
  • 2,352
  • 1
  • 19
  • 27
  • Thank you for the explanation !! Can you please share the links about tez.task.resource.memory.mb? I would like to know how to set it – Ulky Igor Jan 31 '19 at 07:14
  • @UlkyIgor This property needs to be set in tez-site.xml. You can change this property value in that file. Normally tez-site.xml file can be found under TEZ_HOME/conf. Alternatively if you are using Ambari then you can change it from there as well. If you don't see this property there then just add it as below:` tez.task.resource.memory.mb 1024 ` This link will provide some basic description [https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_command-line-installation/content/ref-ffec9e6b-41f4-47de-b5cd-1403b4c4a7c8.1.html] – Jainik Jan 31 '19 at 23:02
  • Thanks you for the answer. Though, I don't have access to the cluster configuration, and just running hive requests via Beeline. I found a way to set it up ;) with --hiveconf – Ulky Igor Feb 04 '19 at 16:10