custom spark does not find hive databases when running on yarn

Question

Starting a custom version of spark on yarn in HDP works fine following the tutorial from https://georgheiler.com/2019/05/01/headless-spark-on-yarn/ i.e. following:

# download a current headless version of spark
export SPARK_DIST_CLASSPATH=$(hadoop classpath)
export HADOOP_CONF_DIR=/usr/hdp/current/spark2-client/conf
export SPARK_HOME=<<path/to>>/spark-2.4.3-bin-without-hadoop/
<<path/to>>/spark-2.4.3-bin-without-hadoop/bin/spark-shell --master yarn --deploy-mode client --queue <<my_queue>> --conf spark.driver.extraJavaOptions='-Dhdp.version=2.6.<<version>>' --conf spark.yarn.am.extraJavaOptions='-Dhdp.version=2.6.<<version>>'

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.3
      /_/

However, a:

spark.sql("show databases").show

only returns:

+------------+
|databaseName|
+------------+
|     default|
+------------+

Now trying to pass the original HDP configuration (which is apparently not read in by my custom version of spark) like:

one:

--files /usr/hdp/current/spark2-client/conf/hive-site.xml

two:

--conf spark.hive.metastore.uris='thrift://master001.my.corp.com:9083,thrift://master002.my.corp.com:9083,thrift://master003.my.corp.com:9083' --conf spark.hive.metastore.sasl.enabled='true' --conf hive.metastore.uris='thrift://master001.my.corp.com:9083,thrift://master002.my.corp.com:9083,thrift://master003.my.corp.com:9083' --conf hive.metastore.sasl.enabled='true'

three:

--conf spark.yarn.dist.files='/usr/hdp/current/spark2-client/conf/hive-site.xml'

four:

--conf spark.sql.warehouse.dir='/apps/hive/warehouse'

all does not help to solve the issue. How can I get spark to recognize the hive databases?

Answer 1

You can copy the hive-site.xml located in the /usr/hdp/hdp.version/hive/conf or /opt/hdp/hdp.version/hive/conf, depending upon where the HDP is installed, in to the conf directory of the headless spark installation. Now when you restart the Spark-Shell it should pick this hive configuration and load all the schemas present in Apache Hive

Answer 2

Hive jars need to be in the classpath of Spark for hive support to be enabled. if the hive jars are not present in classpath, the catalog implementation used is in-memory
In spark-shell we can confirm this by executing

sc.getConf.get("spark.sql.catalogImplementation") 

which will give in-memory

Why are hive classes required

    def enableHiveSupport(): Builder = synchronized {
      if (hiveClassesArePresent) {
        config(CATALOG_IMPLEMENTATION.key, "hive")
      } else {
        throw new IllegalArgumentException(
          "Unable to instantiate SparkSession with Hive support because " +
            "Hive classes are not found.")
      }
    }

SparkSession.scala

  private[spark] def hiveClassesArePresent: Boolean = {
    try {
      Utils.classForName(HIVE_SESSION_STATE_BUILDER_CLASS_NAME)
      Utils.classForName("org.apache.hadoop.hive.conf.HiveConf")
      true
    } catch {
      case _: ClassNotFoundException | _: NoClassDefFoundError => false
    }
  }

If the classes are not present, Hive support is not enabled. Link to the code where the above checks happen as part of Spark shell initialization.

In the above code pasted as part of question, SPARK_DIST_CLASSPATH is populated only with the Hadoop classpath and the paths to Hive jars missing.