As currently it is not possible to create a shortcut in oneLake for private enpoint enabled azure storage so I am exploring another way to get the data from secure datalake to oneLake. According to MS Documentation, It is possible to integrate azure databricks with onelake so I am reading the data from pe enabled adls and trying to write it to oneLake. While writing to oneLake, I am facing below error:
Operation failed: "Forbidden", 403, HEAD, https://onelake.dfs.fabric.microsoft.com/d36-47c2-83e3-676eec7d9/22f90e-f0d9-4559008060a/Files/RAW
I have used below code for it.
df_zipped = spark.read.format("parquet").option("compression", "gzip").option("header", True).load("/mnt/defined/measuremts-2019.gz.parquet")
oneLake = 'https://onelake.dfs.fabric.microsoft.com/d36-47c2-83e3-676eec7d9/22f90e-f0d9-4559008060a/Files/RAW'
df_zipped.write.format("csv").option("header", "true").mode("overwrite").csv(oneLake)
I understand that this is authorization issue so can any one of you please help me to understand how can I authenticate to oneLake from azure databricks.
I am accessing fabric from https://app.fabric.microsoft.com/ using Azure AD authentication so I am not sure how can I authenticate my identity from azure databricks to onelake.
Please advise.
Thanks
