5

I've been running some hive scripts on an aws emr 4.8 cluster with hive 1.0 and tez 0.8.

My configurations look like this: 

SET hive.exec.compress.output=true;
SET mapred.output.compression.type=BLOCK;
SET hive.exec.dynamic.partition = true;
SET hive.exec.dynamic.partition.mode = nonstrict;
set hive.execution.engine=tez;
set hive.merge.mapfiles=false;
SET hive.default.fileformat=Orc;
set tez.task.resource.memory.mb=5000;
SET hive.tez.container.size=6656;
SET hive.tez.java.opts=-Xmx5120m;
set hive.optimize.ppd=true;

And my global configs are:

hadoop-env.export   HADOOP_HEAPSIZE 4750
hadoop-env.export   HADOOP_DATANODE_HEAPSIZE    4750
hive-env.export HADOOP_HEAPSIZE 4750

While running my script, I get the following error:

Container [pid=19027,containerID=container_1477393351192_0007_02_000001] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 1.9 GB of 5 GB virtual memory used. Killing container.

On googling this error, I read that set tez.task.resource.memory.mb will change the physical memory limit, but clearly I was mistaken. What am I missing?

jackStinger
  • 2,035
  • 5
  • 23
  • 36

4 Answers4

7

I have had this problem a lot. The changing 

Set hive.tez.container.size=6656;
Set hive.tez.java.opts=-Xmx4g;

does not fix the problem for me but this does: 

set tez.am.resource.memory.mb=4096;
Joe S
  • 410
  • 6
  • 16
  • `tez.am.resource.memory.mb=4096;` is the solution for me. in my case it's the app master (am) has too little memory so the job failed when scheduling. – StanleyZ Nov 26 '19 at 17:15
2

Set the Tez container size to be a larger multiple of the YARN container size (4GB):

SET hive.tez.container.size=4096MB

"hive.tez.container.size" and "hive.tez.java.opts" are the parameters that alter Tez memory settings in Hive. If "hive.tez.container.size" is set to "-1" (default value), it picks the value of "mapreduce.map.memory.mb". If "hive.tez.java.opts" is not specified, it relies on the "mapreduce.map.java.opts" setting. Thus, if Tez specific memory settings are left as default values, memory sizes are picked from mapreduce mapper memory settings "mapreduce.map.memory.mb".

https://documentation.altiscale.com/memory-settings-for-tez

For more info Tez configuration and Tez memory tuning

Note: Set in MB with Ambari

BruceWayne
  • 3,286
  • 4
  • 25
  • 35
  • Is there a reason SET hive.tez.container.size=6656; won't work? – jackStinger Oct 26 '16 at 05:33
  • Also, Query returned non-zero code: 1, cause: 'SET hive.tez.container.size=6656MB' FAILED because hive.tez.container.size expects INT type value. – jackStinger Oct 26 '16 at 05:43
  • So I had set the container size, as well as the tez.java.opts in my configurations already. Despite that, my physical memory is 1G. The altiscale page is exactly where I found these, but didn't work for me. :/ – jackStinger Oct 26 '16 at 08:57
  • does this commands work on hadoop conf? or just for hive? should this works for example (set mapreduce.reduce.memory.mb = 4096)? – Reihan_amn Nov 19 '18 at 06:53
1

Incase anyone else stumbles upon this thread trying to solve this above, here is a link to a real solution that worked for me where all the other solutions did not.

http://moi.vonos.net/bigdata/hive-cli-memory/

TL;DR add these to your hive call --hiveconf tez.am.resource.memory.mb=<size as int> --hiveconf tez.am.launch.cmd-opts=""

Alex
  • 280
  • 3
  • 18
0
Set hive.tez.container.size=6656
Set hive.tez.java.opts=-Xmx4g
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
curiouslever
  • 110
  • 1
  • 10
  • 2
    SO how is it different from SET hive.tez.container.size=6656; SET hive.tez.java.opts=-Xmx5120m; – jackStinger Oct 26 '16 at 04:08
  • There are no semicolons at the end of the statements. I had a similar failure "... expects INT type value" with container.size. I removed the semicolon at the end of the statement and the failure went away. But I have no idea why that fix works. – user3062149 May 10 '18 at 20:30
  • does this commands work on hadoop conf? or just for hive? should this works for example (set mapreduce.reduce.memory.mb = 4096)? – Reihan_amn Nov 19 '18 at 07:08