0

Is it possible to mount ADLS Gen2 account/container without configuring service principle credentials to access storage?

The following article states that it's possible to access a ADLS storage account with first configuring a service principle.

https://learn.microsoft.com/en-us/azure/databricks/data-governance/credential-passthrough/adls-passthrough

I normally mount a drive in Databricks by first creating a service principle by Registering an Azure AD application using App Registrations. I would use the following code to mount:

if check(mount)==1:
  resultMsg = "<div>%s is already mounted. </div>" % mount
else:
  dbutils.fs.mount(
  source = f"abfss://root@{Lake}.dfs.core.windows.net/",
  mount_point = mount,
  extra_configs = configs)
  resultMsg = "<div>%s was mounted. </div>" % mount

At present I am waiting for the administrator at my place of work to give me the permissions to register applications.

So, I was wondering if I could mount a drive from Databricks without first registering an application?

I tried @JayashankarGS suggestion as follows:

mount = "/mnt/lake"

if check(mount)==1:
  resultMsg = "<div>%s is already mounted. </div>" % mount
else:
  dbutils.fs.mount(
  source = f"abfss://root@{Lake}.dfs.core.windows.net/",
  mount_point = mount,
  extra_configs = configs)
  resultMsg = "<div>%s was mounted. </div>" % mount

But I got the error;

IllegalArgumentException: Unsupported Azure Scheme: abfss
Patterson
  • 1,927
  • 1
  • 19
  • 56
  • You can mount using account key. – JayashankarGS May 22 '23 at 11:23
  • Hi @JayashankarGS, did you mean something like this: ```mount = "/mnt/lake" if check(mount)==1: resultMsg = "
    %s is already mounted.
    " % mount else: dbutils.fs.mount( source = "abfss://root@adlspretbiukadlsdev.blob.core.windows.net", mount_point = mount, extra_configs = {"fs.azure.account.key.adlspretbiukadlsdev.blob.core.windows.net":""})```     – Patterson May 22 '23 at 12:10
  • Hi @JayashankarGS, thanks for reaching out. I have updated the question with your suggestion, but it didn't work. Do you have an example of your suggestion? – Patterson May 22 '23 at 12:14
  • is it ok using the url like `abfss://root@adlspretbiukadlsdev.blob.core.windows.net/dir` instead of mounting? – JayashankarGS May 22 '23 at 12:21

2 Answers2

1

You can try to mount storage using account key

I did similar thing in my project as well!!

You can use below code as a reference

    Container_name = "root"
    storage_account = "lake"
    key = ""

    url = "wasbs://" + container_name + "@" + storage_account + ".blob.core.windows.net/"
    config = "fs.azure.account.key." + storage_account + ".blob.core.windows.net"

    mount_folder = "/mnt/lake"
    mounted_list = dbutils.fs.mounts()

    mounted_exist = False
    for item in mounted_list:
        if mount_folder in item[0]:
            mounted_exist = True
            break

    if not mounted_exist:
        dbutils.fs.mount(source = url, mount_point = mount_folder, extra_configs = {config : key})
Adriaan
  • 17,741
  • 7
  • 42
  • 75
Bhagyashree
  • 28
  • 3
  • Hi @Bhagyashree, thanks for reaching out. You sample code appears to work. But I don't undersstand why your mount looks like ```/mnt/lake wasbs://root@adlspretbiukadlsdev.blob.core.windows.net/``` whereas when I mount looks like the following ```/mnt/lake abfss://root@prodlakeaccuks01.dfs.core.windows.net/```. Surely, its because your code mounts a BLOB store, i.e. wasp. Whereas my code mounts ADLS Gen 2 ie. abfss. Or am i incorrect? – Patterson May 22 '23 at 20:24
  • Sorry, I meant my mount looks like the following ```/mnt/lake abfss://root@adlspretbiukadlsdev.dfs.core.windows.net/``` whereas your mount looks like the following ```/mnt/lake wasbs://root@adlspretbiukadlsdev.blob.core.windows.net/``` – Patterson May 22 '23 at 20:40
  • Hi @Patterson, Sorry for the late reply!! abfss and wasbs are 2 ways by which you can access Azure Storage account, only catch is wasbs is legacy method to do it. You can check out this one - https://stackoverflow.com/questions/60277545/what-is-the-difference-between-abfss-and-wasbs-in-azure-storage – Bhagyashree May 25 '23 at 03:13
0

You use storage account key to access data in adls2. Below is code for it.

    spark.conf.set("fs.azure.account.key.<storage account name>.dfs.core.windows.net","Your stoarge account key")
    dbutils.fs.ls("abfss://<container name>@<storage account name>.dfs.core.windows.net/<dir>")

Output:

enter image description here

Or you can follow below approach.

    dbutils.fs.mount(source = "wasbs://data@jgsadls.blob.core.windows.net",
            mount_point = "/mnt/iotdata",
            extra_configs = {"fs.azure.account.key.jgsadls.blob.core.windows.net":"Your account key"})
    dbutils.fs.ls("/mnt/iotdata/json_data/")

This is the result.

enter image description here

JayashankarGS
  • 1,501
  • 2
  • 2
  • 6
  • Hi @JayashankarGS, thanks for getting in touch. I would like to follow you alternative approach, but in that approach you're mounting a BLOB i.e wasbs account, as opposed to an ADLS gen2 account abfss, correct? – Patterson May 22 '23 at 15:21
  • No that is adls gen2 storage account. I enabled hierarchical namespace and in my first approach you can see i used abfss protocal to list down files with same storage account `jgsadls`and container `data`. – JayashankarGS May 22 '23 at 15:44