1

I have installed Hadoop,Hive,Spark in Ubuntu 18 LTS. I am trying to find out how to connect the spark with hive tables. There needs to be some link between the HIVE and SPARK, for Spark to recognize the tables created in Hive and doing a computation on them.

Any help on this is highly appreciated. RIght now it says table not found when i run

spark.sql("select * from dummy.table_name")

Mohanram Krishnan
  • 145
  • 1
  • 12
  • try creating spark session including `.enableHiveSupport()` EX: **`val spark = SparkSession.builder().appName("Spark Hive Example").enableHiveSupport().getOrCreate()`** – notNull Mar 14 '20 at 00:41
  • ... and do u have **`dummy`** database in hive already? – notNull Mar 14 '20 at 00:42
  • Yes i have a dummy database. I have tried the enableHiveSupport() option, but even before that there needs to be a xml level property to point the HIVE directory to spark. I am trying to find that one. – Mohanram Krishnan Mar 14 '20 at 00:44
  • 1
    Copy your `hive-site.xml` into spark `conf/` directory then restart spark.. it should work then.. – notNull Mar 14 '20 at 00:46
  • It throws, ```Caused by: java.sql.SQLException: Failed to start database '/home/max/apache-hive-3.1.2-bin/metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@1635b03f, see the next exception for details.``` – Mohanram Krishnan Mar 14 '20 at 00:51
  • Caused by: ERROR XSLAN: Database at /home/max/apache-hive-3.1.2-bin/metastore_db has an incompatible format with the current version of the software. The database was created by or upgraded by version 10.14. – Mohanram Krishnan Mar 14 '20 at 00:51
  • @Shu migrated to mySql and it works now. Thanks for your assistance! – Mohanram Krishnan Mar 14 '20 at 03:05
  • Does this answer your question? [How to connect to remote hive server from spark](https://stackoverflow.com/questions/39997224/how-to-connect-to-remote-hive-server-from-spark) – Michael Heil Mar 14 '20 at 10:08
  • All i did was to copy the hive-site.xml from hive/conf to spark/conf folder. and I also migrated from Derby to MySQL database + copying the mysql connector.jar to spark. – Mohanram Krishnan Mar 16 '20 at 19:21

0 Answers0