1

Please can you help me understand what is 512mb (highlighed in bold) in Hive context? Which memory size it is? I have set export HADOOP_HEAPSIZE=4192 in my hive-site.xml file

hadoop@master:~/hive/conf$ ps -ef | grep 'hive'

hadoop 5587 1 0 Feb14 ? 00:05:27 /usr/lib/jvm/default-jdk/bin/java -Xmx4192m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/hadoop-2.7.3/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/hadoop/hadoop-2.7.3 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop-2.7.3/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx512m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /home/hadoop/hive/lib/hive-metastore-2.3.2.jar org.apache.hadoop.hive.metastore.HiveMetaStore

hadoop 9903 8034 0 10:54 pts/0 00:00:00 grep --color=auto hive

hadoop 21646 15918 1 07:37 pts/3 00:03:02 /usr/lib/jvm/default-jdk/bin/java -Xmx4192m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/hadoop-2.7.3/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/hadoop/hadoop-2.7.3 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop-2.7.3/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx512m -Dproc_hivecli -Dlog4j.configurationFile=hive-log4j2.properties -Djava.util.logging.config.file=/home/hadoop/hive/conf/parquet-logging.properties -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /home/hadoop/hive/lib/hive-cli-2.3.2.jar org.apache.hadoop.hive.cli.CliDriver

Arun
  • 651
  • 2
  • 7
  • 21

1 Answers1

2

The variable you edited is for the clients, not the servers, and you don't export anything from the XML files

To configure the heap size for HiveServer2 and Hive metastore, set the -Xmx parameter in the HADOOP_OPTS variable to the desired maximum heap size in /etc/hive/hive-env.sh

VS

To configure the heap size for the Beeline CLI, set the HADOOP_HEAPSIZE environment variable in /etc/hive/hive-env.sh

Besides that, both values made it though, so you might want to look at

Duplicated Java runtime options : what is the order of preference?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245