2

I have kerbores enabled Cloudera cluster. Spark can access Hive table when I use client deployment mode.

I executed kinit command and then executed spark2-submit. Spark can access the Hive table when I use client mode.

spark2-submit --master yarn --deploy-mode client  --keytab XXXXXXXXXX.keytab --principal XXXXXXXXXXX@USER.COM  --conf "spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:MaxPermSize=1024M -Djava.security.krb5.conf=/etc/krb5.conf" test.jar

But when I use cluster mode, spark gives table not found error.

spark2-submit --master yarn --deploy-mode cluster  --keytab XXXXXXXXXX.keytab --principal XXXXXXXXXXX@USER.COM  --conf "spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:MaxPermSize=1024M -Djava.security.krb5.conf=/etc/krb5.conf" test.jar
lucy
  • 4,136
  • 5
  • 30
  • 47
  • 3
    Possible duplicates of https://stackoverflow.com/questions/34034488/hive-tables-not-found-when-running-in-yarn-cluster-mode – Vin Mar 18 '19 at 18:14
  • See also https://stackoverflow.com/questions/45477155/missing-hive-site-when-using-spark-submit-yarn-cluster-mode > injecting Metastore address in `spark.hadoop.hive.metastore.uris` directly, or using `spark.yarn.dist.files` to specify path to hive-site.xml – Samson Scharfrichter Mar 19 '19 at 11:54

0 Answers0