3

In databricks runtime version 6.6 I am able to successfully run a shell command like the following:

%sh ls /dbfs/FileStore/tables

However, in runtime version 7, this no longer works. Is there any way to directly access /dbfs/FileStore in runtime version 7? I need to run commands to unzip a parquet zip file in /dbfs/FileStore/tables. This used to work in version 6.6 but databricks new "upgrade" breaks this simple core functionality.

Not sure if this matters but I am using the community edition of databricks.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Willard
  • 502
  • 8
  • 21

3 Answers3

2

WHen you run %sh ls /dbfs/FileStore/tables you can't Access /dbfs/FileStore using shell commands in databricks runtime version 7 because by default, the folder named '/dbfs/FileStore' does not exists in the 'dbfs'.

enter image description here

Try to upload some files in '/dbfs/FileStore/Tables'.

enter image description here

Now, try to run the same command again %sh ls /dbfs/FileStore/tables, now you see the results because we have upload the data into /dbfs/FileStore/tables folder.

enter image description here

CHEEKATLAPRADEEP
  • 12,191
  • 1
  • 19
  • 42
  • If my answer is helpful for you, you can [accept it as an answer](https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work). This can be beneficial to other community members. Thank you. – CHEEKATLAPRADEEP Sep 09 '20 at 12:02
  • 1
    This is incorrect answer because it was not done on Community Edition that is different from Azure Databricks – Alex Ott Aug 27 '21 at 20:01
2

The /dbfs mount doesn't work on Community Edition with DBR >= 7.x - it's a known limitation.

You can workaround this limitation by working with files on the driver node and upload or download files using the dbutils.fs.cp command (docs). So your code will look as following:

#write a file to local filesystem using Python I/O APIs
...
# upload file to DBFS
dbutils.fs.cp('file:/tmp/local-path', 'dbfs:/FileStore/tables/dbfs_file.txt')

and reading from DBFS will look as following:

# copy file from DBFS to local file_system
dbutils.fs.cp('dbfs:/tmp/dbfs_file.txt', 'file:/tmp/local-path')
# read the file locally
...
Alex Ott
  • 80,552
  • 8
  • 87
  • 132
0

I know this question is a year old, but I wanted to share other posts that I found helpful in case someone has the same question.

I found the comments in this similar question to be helpful: How to access DBFS from shell?. The comments in the aforementioned post, also references Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: [Errno 2] No such file or directory: which I found helpful as well.

I learned in Community Edition ls /dbfs/FileStore/tables is not possible because the dbfs itself is not mounted on the nodes and the feature is disabled.

Rick D.
  • 1
  • 1