6

I am trying to connect to abfss directly(without mounting to DBFS) and trying to open json file using open() method in databricks.

json_file = open("abfss://@.dfs.core.windows.net/test.json') databricks is unable to open file present in azure blob container and getting below error : FileNotFoundError: [Errno 2] No such file or directory: 'abfss://@.dfs.core.windows.net/test.json'

I have done all the configuration setting using service principal. Please suggest other way of opening file using abfss direct path.

Deepika
  • 61
  • 1
  • 2

1 Answers1

6

the open method works only with local files - it doesn't know anything about abfss or other cloud storages. You have following choice:

  1. use dbutils.fs.cp to copy file from ADLS to local disk of driver node, and then work with it, like: dbutils.fs.cp("abfss:/....", "file:/tmp/my-copy")
  2. Copy file from ADLS to driver node using the Azure SDK

The first method is easier to use than second

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • Thanks @Alex Ott, I will try to implement this. – Deepika Apr 10 '21 at 13:45
  • This is such a great solution that is far better than using sas token urls to access files. I used this with openpyxl to extract sheets from xlsx files in synapes notebooks. Before your solution, I kept getting 'file not found' errors. openpyxl needs the local file system. Copying from ADLS to local is perfect for my solution. Thank you. – user3486773 Aug 24 '23 at 14:38