1

I have a .sh file from which I am passing values to .hql, but its giving me errors

sm=1
XXXXX=""
while read -r line
do
   name="$line"
   XXXXX="hive$name(${XXXX[$sm]%?})"
   echo $XXXXX
   hive -hiveconf var1=$XXXXX -hiveconf var2=/user/cloudera/project -hiveconf var3=$name -f test1.hql
   sm=$((sm + 1))
done < "$filename"


CREATE EXTERNAL TABLE IF NOT EXISTS ${hiveconf:var1}
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
location ${hiveconf:var2/hiveconf:var3};

*Please note that $XXXXX is creating a tablename with schema after reading from the file and some logic. When I echo it, there is no problem, but problem comes in .hql file. Error is somewhat like below :

at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

FAILED: ParseException line 2:4 cannot recognize input near 'ROW' 'FORMAT' 'DELIMITED' in column type WARN: The method class org.apache.commons.logging.impl.SLF4JLogFactory#release() was invoked. WARN: Please see http://www.slf4j.org/codes.html#release for an explanation.

Gurpreet Singh
  • 67
  • 1
  • 8
  • Variable XXXXX contains "hivecategories(category_id int,category_department_id int,category_name string)"... but when I echo it on hql, it isn;t passing the whole value... just "hivecategories(category_id " and that is causing the error – Gurpreet Singh Sep 06 '18 at 18:43

2 Answers2

1

Try to quote variable:

hive -hiveconf var1="$XXXXX"

All such variables should be quoted

Use this command inside script to check value passed:

! echo "${hiveconf:var1}";
leftjoin
  • 36,950
  • 8
  • 57
  • 116
1

An alternate to above is using hivevar:

sm=1
XXXXX=""
while read -r line
do
   name="$line"
   XXXXX="hive$name(${XXXX[$sm]%?})"
   echo "${XXXXX}"
   hive -hivevar var1=${XXXXX} -hivevar var2="/user/cloudera/project" -hivevar var3=${name} -f test1.hql
   sm=$((sm + 1))
done < "$filename"


CREATE EXTERNAL TABLE IF NOT EXISTS ${var1}
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
location ${var2/var3};

Also here is a link for a read on the difference between hiveconf and hivevar in case the curiosity bug bites :)

What is the difference between -hivevar and -hiveconf?

Ishan Sharma
  • 53
  • 7