0

I am testing Hive on Spark as engine, but I want at the same time, use Hive metastore to access schemas using Spark, but till now, every combination of versions I have tested have failed. Could you please recommend to me one version of Spark and one version of Hive that I can use to achieve this? Thank you

Edisnel C. C.
  • 1
  • What does `tested` mean? Could you walk us through? – Bala Sep 28 '18 at 05:10
  • I want to install Spark 2.3.1(or at least 2.2.0,the version I have on production) with a version of Hive to read schemas in its metastore, but when I achieve it, then Hive on Spark as execution engine (hive.execution.engine=spark) is not working due to JavaSparkListener ERROR. In Spark source code there are restrictions on Hive (0.12.0 through 2.1.1). I can not have both at once? Thank you – Edisnel C. C. Sep 28 '18 at 13:25
  • 1
    Install a distro (e.g. latest HortonWorks HDP) where components have been aligned to work together. Building your own mix is hard... – Samson Scharfrichter Sep 28 '18 at 18:23
  • https://stackoverflow.com/questions/41953688/setting-spark-as-default-execution-engine-for-hive – Bala Sep 28 '18 at 19:20
  • Ok, thank you very much for your answers – Edisnel C. C. Sep 28 '18 at 19:54
  • Does this answer your question? [How to connect Spark SQL to remote Hive metastore (via thrift protocol) with no hive-site.xml?](https://stackoverflow.com/questions/31980584/how-to-connect-spark-sql-to-remote-hive-metastore-via-thrift-protocol-with-no) – Jacek Laskowski Mar 22 '20 at 16:49

0 Answers0