wait until the blob storoage folder is created

Question

I would like to download a picture into a blob folder. Before that I need to create the folder first. Below codes are what I am doing. The issue is the folder needs time to be created. When it comes to with open(abs_file_name, "wb") as f: it can not find the folder. I am wondering whether there is an 'await' to get to know the completion of the folder creation, then do the write operation.

for index, row in data.iterrows():
    url = row['Creatives']
    file_name = url.split('/')[-1]
    r = requests.get(url)
    abs_file_name = lake_root + file_name
    dbutils.fs.mkdirs(abs_file_name)
    if r.status_code == 200:
        with open(abs_file_name, "wb") as f:
            f.write(r.content)

Is this regular Blob Storage or ADLS Gen2? There is a significant difference, one of the most important of which is that folders are not actually defined in blob storage. This answer may give you some insights: https://stackoverflow.com/questions/65912473/how-to-create-directories-in-azure-storage-container-without-creating-extra-file/65923129#65923129 — Joel Cochran, Jan 05 '23 at 16:22

score 0 · Answer 1 · answered Jan 06 '23 at 04:57

The final sub folder will not be created when using dbutils.fs.mkdirs() on blob storage.
It creates a file with the final sub folder name which would be considered as a directory, but it is not a directory. Look at the following demonstration:

dbutils.fs.mkdirs('/mnt/repro/s1/s2/s3.csv')

enter image description here

When I try to open this file, the error says that this is a directory.

enter image description here

This might be the issue with the code. So, try using the following code instead:

for index, row in data.iterrows():
    url = row['Creatives']
    file_name = url.split('/')[-1]
    r = requests.get(url) 
    abs_file_name = lake_root + 'fail'  #creates the fake directory (to counter the problem we are facing above)
    dbutils.fs.mkdirs(abs_file_name)
    if r.status_code == 200: 
        with open(lake_root + file_name, "wb") as f: 
            f.write(r.content)

wait until the blob storoage folder is created

1 Answers1