0

I'm using DataBricks Python.

If I save this file

y_pred.to_csv("y_pred.csv")

it works but I don't know where the file is and I can't find it.

When I try to copy the file in another location

dbutils.fs.cp("/y_pred.csv", 'wasbs://...........')

I get the following error:

java.io.FileNotFoundException: /y_pred.csv

Dario Federici
  • 1,228
  • 2
  • 18
  • 40
  • 1
    Specify the file path while saving `df.to_csv(r'/dir/y_pred.csv')` Reference: [CSV File Reading and Writing](https://docs.python.org/3/library/csv.html) and [Save results to csv file with Python](https://stackoverflow.com/questions/3345336/save-results-to-csv-file-with-python) – Ecstasy Jan 20 '22 at 09:52
  • Does this answer your question? [How to save and download locally csv in DBFS?](https://stackoverflow.com/questions/58288111/how-to-save-and-download-locally-csv-in-dbfs) – Ecstasy Jan 20 '22 at 10:01

2 Answers2

0

Hi This is the solution:

dbutils.fs.put('shared_uploads/...........com/y_pred.csv', pd.DataFrame(y_pred).to_csv())
Dario Federici
  • 1,228
  • 2
  • 18
  • 40
0

Based on the databricks concepts , If data frame fits in a driver memory and you want to save to local files system ,that time we can use to_csv. So your file will be stored in driver node . When you are reading the drive node local file through dbutils.fs we should add (file:/<filelocation). So try to read it like dbutils.fs.cp("file:/y_pred.csv", 'wasbs://...........') will it work . You can understand on dbfs location ref

Karthikeyan Rasipalay Durairaj
  • 1,920
  • 13
  • 35