1

I have a CSV file in my blob storage. I want to download the file. The weird things is, I can't actually see the file. The file CSV is created by a Python job and converted into a Scala dataframe.

When I run these 2 lines of code:

# convert python df to spark df and export the spark df
spark_df = spark.createDataFrame(df)
## Write Frame out as Table
spark_df.write.csv("dbfs:/rawdata/corp/AAA.csv")

I get this error: 

org.apache.spark.sql.AnalysisException: path dbfs:/rawdata/corp/AAA.csv already exists.;

The weird thing is that I can't see the file when I'm using Azure Storage Explorer. Apparently the file exists, even though I can't see it. How can I download this CSV file? I would like to use Databricks, preferably, or maybe something else, if someone here can suggest a better option.

Thanks.

ASH
  • 20,759
  • 19
  • 87
  • 200

3 Answers3

2

How about using the blob storage sdk and the following command

# Download as a file
block_blob_service.get_blob_to_path(container_name, blob_name, local_file_name)
kgalic
  • 2,441
  • 1
  • 9
  • 21
2

Note: Using GUI, you can download full results (max 1 millions rows).

enter image description here

To download full results (more than 1 million), first save the file to dbfs and then copy the file to local machine using Databricks cli as follows.

dbfs cp "dbfs:/FileStore/tables/AA.csv" "A:\AzureAnalytics"

Reference: Databricks file system

The DBFS command-line interface (CLI) uses the DBFS API to expose an easy to use command-line interface to DBFS. Using this client, you can interact with DBFS using commands similar to those you use on a Unix command line. For example:

# List files in DBFS
dbfs ls
# Put local file ./apple.txt to dbfs:/apple.txt
dbfs cp ./apple.txt dbfs:/apple.txt
# Get dbfs:/apple.txt and save to local file ./apple.txt
dbfs cp dbfs:/apple.txt ./apple.txt
# Recursively put local dir ./banana to dbfs:/banana
dbfs cp -r ./banana dbfs:/banana

Reference: Installing and configuring Azure Databricks CLI

Hope this helps.

CHEEKATLAPRADEEP
  • 12,191
  • 1
  • 19
  • 42
  • You are exactly right. I just figured this out a few days back. Thanks for sharing. – ASH Oct 27 '19 at 14:32
1

I found another nice solution here.

https://docs.databricks.com/notebooks/notebooks-use.html

enter image description here

Just before this step, do display the contents of a dataframe, run this line of code.

display(df)
ASH
  • 20,759
  • 19
  • 87
  • 200