I have written a function to upload a .csv file from a local file system to Azure Blob storage. The function is being used to upload four csv files. Three of the files upload without any issues (their size ranges from 3 to 75Mb); however one file (the third of the four files) raises the following exception - The operation did not complete (write) (_ssl.c:2158)
This questionable file is able to be uploaded manually from the local file system to the storage container without any issue, so there is no question of the file being inappropriate or corrupted.
The function is given below:
def export_to_blob(source_path,source_file_name):
try:
connect_str = os.getenv('AZURE_STORAGE_CONNECTION_STRING')
# Create the BlobServiceClient object which will be used to create a container client
blob_service_client = BlobServiceClient.from_connection_string(connect_str)
container_name = 'data-warehouse-blob'
# Define the container
container_client = blob_service_client.get_container_client(container_name)
# Create a file in local data directory to upload and download
local_path = source_path
local_file_name = source_file_name
upload_file_path = os.path.join(local_path, local_file_name)
# Create a blob client using the local file name as the name for the blob
blob_client = blob_service_client.get_blob_client(container=container_name, blob=local_file_name)
print("\nUploading to Azure Storage as blob:\n\t" + local_file_name)
# Upload the created file
with open(upload_file_path, "rb") as data:
blob_client.upload_blob(data,overwrite=True)
except Exception as ex:
print('Exception:')
print(ex)
Any explination of this rather cryptic exception message would be gratefully received.
EDIT: I've enabled logging and the log details are as follows:
Request URL: 'https://orrblobstorage001.blob.core.windows.net/data-warehouse-blob/DW_327_RELIABILITY.csv'
Request method: 'PUT'
Request headers:
'Content-Type': 'application/octet-stream'
'Content-Length': '48570614'
'x-ms-version': '2019-07-07'
'x-ms-blob-type': 'BlockBlob'
'x-ms-date': 'Mon, 13 Jul 2020 13:33:00 GMT'
'x-ms-client-request-id': '5ed935fa-c50d-11ea-8108-9061ae394c51'
'User-Agent': 'azsdk-python-storage-blob/12.3.2 Python/3.6.8 (Windows-10-10.0.17134-SP0)'
'Authorization': '*****'
Request body:
b'Base_Location,TOC,Criticality,Location,Location_Type,Natural_Frequency,Data_Type,Option_1,Option_2,Option_3,Option_4,Option_5,min_value,max_value,Date,value\r\n
[[snipped the data returned here]]
DU_Two,MDU_Criticality,Period,Service_Affecting_Failures,Track,Track failures (service affecting),x,x,x,2.0,19.0,2020-05-30,2.0\r\n'
When I reduced the amount of data being uploaded from 50MB to 35 MB the upload works without any issue, but other files at 75MB are being uploaded without issue. The DEBUG log is as follows. It appears that the first log simply dies without any warning or error message, but the smaller file gets a response.
What is going on here?
Request URL: 'https://orrblobstorage001.blob.core.windows.net/data-warehouse-blob/DW_327_RELIABILITY.csv'
Request method: 'PUT'
Request headers:
'Content-Type': 'application/octet-stream'
'Content-Length': '36368360'
'x-ms-version': '2019-07-07'
'x-ms-blob-type': 'BlockBlob'
'x-ms-date': 'Mon, 13 Jul 2020 13:53:50 GMT'
'x-ms-client-request-id': '481fbd0c-c510-11ea-bf7c-9061ae394c51'
'User-Agent': 'azsdk-python-storage-blob/12.3.2 Python/3.6.8 (Windows-10-10.0.17134-SP0)'
'Authorization': '*****'
Request body:
b'Base_Location,TOC,Criticality,Loc
[[snipped the data returned here]]
x,x,x,2.0,17.0,2020-05-30,2.0\r\n'
Response status: 201
Response headers:
'Content-Length': '0'
'Content-MD5': 'AVRkDNBgBIdL5/bgiJQfgg=='
'Last-Modified': 'Mon, 13 Jul 2020 13:54:08 GMT'
'ETag': '"0x8D8273436B6BA9F"'
'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
'x-ms-request-id': '85868dcc-a01e-0092-031d-5926f6000000'
'x-ms-client-request-id': '481fbd0c-c510-11ea-bf7c-9061ae394c51'
'x-ms-version': '2019-07-07'
'x-ms-content-crc64': 'YgACieEqdpI='
'x-ms-request-server-encrypted': 'true'
'Date': 'Mon, 13 Jul 2020 13:54:07 GMT'
Response content:
