2

So when I was trying to work with some ideas on DataBricks Community Edition today, I suddenly find out the python pandas can no longer read a existing uploaded table. The directories were working before, but none of my previously uploaded tables can be read any more, all return in "File ... Not Exist". Anyone know what we can do?

Command used:

import pandas as pd
df = pd.read_csv('/dbfs/FileStore/tables/iris.csv')

File is definitely there, this also affects all my other previously uploaded files.

DBFS

enter image description here

Any suggestion would be appreciated

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
stan12
  • 31
  • 4
  • 2
    Does this answer your question? [Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: \[Errno 2\] No such file or directory:](https://stackoverflow.com/questions/63552922/not-able-to-cat-dbfs-file-in-databricks-community-edition-cluster-filenotfounde) – Alex Ott Feb 22 '21 at 08:27

1 Answers1

1

How did you load the csv? from blob? if from blob you probably have not mounted.

Can you try read as spark? If you can't read then the file is corrupt. If you can, try apply sdf.to.pandas().

sdf = spark.read.csv('/FileStore/tables/iris.csv', header="true", inferSchema="true")
wwnde
  • 26,119
  • 6
  • 18
  • 32
  • 1
    it's a community edition, there are limitations there. see the answer that I linked... – Alex Ott Feb 22 '21 at 08:28
  • 1
    Hi, thanks for the solution, so it works with spark. I am using DBR 7.6, so I will need to find a way to deal with this. – stan12 Feb 23 '21 at 04:48
  • One more question, how do I mount the "/FileStore/" so that I can access it like before? Thanks – stan12 Feb 23 '21 at 05:39
  • Try this documentation and let us know if you run into issues https://docs.databricks.com/data/data-sources/azure/azure-storage.html#language-python – wwnde Feb 23 '21 at 05:40