1

We have an HDInsight cluster in our setup and we are storing the data in Hive tables(The data lies as External tables in ADLS and the metadata in the External metastore and accessed using the Hive service from our Azure cluster). What is the best way to share this data with other Azure clusters, not necessarily within the same subscription?

Azure has this concept of Service principals, so we’d need to setup the acls to allow the other cluster’s service principal access to the ADLS folders corresponding to the hive tables that we share. Additionally, how can our cluster’s hiveserver2 url be used as jdbc connection by the other Azure instances, so that they can query the data? What cluster login should we provision for them to be able to use our HiveServer2 to query the data in our Hive tables?

I understand the right way to do this would be to use the Azure ESP service, but that is apparently a costly choice.

Providing them access only to the ADLS folders also seems incorrect as the metadata is then not used for accessing the data...

Happypig375
  • 1,022
  • 1
  • 12
  • 32
user2674874
  • 11
  • 2

1 Answers1

0

The scenario described could be achieved with a shared metastore. As you already have an external metastore, you could use it also for other clusters. Prerequisite is that the Hive-Metastore version for the clusters accessing this metastore is the same.

You can also change the core-site settings in the cluster creation templates, to have the same file system structures (if you don' t use full-qualified storage account names):

"core-site": {
    "fs.defaultFS": "adl://home",
    "dfs.adls.home.hostname": "<your-datalake-store>.azuredatalakestore.net",
    "dfs.adls.home.mountpoint": "/<your-adl-home-mountpoint-dir>",...
     },

In Spark this would be done with:

spark.conf.set("dfs.adls.home.hostname", "<your-datalake-store>.azuredatalakestore.net")
spark.conf.set("dfs.adls.home.mountpoint", "/<your-adl-home-mountpoint-dir>")

See also the following blog-post.

Check also the following answer how to access HDInsight from pyspark (also from "outside").

Hauke Mallow
  • 2,887
  • 3
  • 11
  • 29