Large Fileupload to Azure with Blob Storage with BlobContainerClient

Question

I want to upload large Files to Azure Blob Storage (500-2000MB) and I try to do this with the following code:

private BlobContainerClient containerClient;    
public async Task<UploadResultDto> Upload(FileInfo fileInfo, string remotePath)
        {
            try
            {
                var blobClient = containerClient.GetBlobClient(remotePath + "/" + fileInfo.Name);

                var transferOptions = new StorageTransferOptions
                {
                    MaximumConcurrency = 1,
                    MaximumTransferSize = 10485760,
                    InitialTransferSize = 10485760
                };

                await using var uploadFileStream = File.OpenRead(fileInfo.FullName);
                
                await blobClient.UploadAsync(uploadFileStream, transferOptions: transferOptions);
                uploadFileStream.Close();

                return new UploadResultDto()
                {
                    UploadSuccessfull = true
                };

            }
            catch (Exception ex)
            {
                Log.Error(ex,$"Error while uploading File {fileInfo.FullName}");
            }

            return new UploadResultDto()
            {
                UploadSuccessfull = false
            };
        }

I instantly get the following message:

The specified blob or block content is invalid.
RequestId:c5c2d925-701e-0035-7ce0-8691a6000000
Time:2020-09-09T19:33:40.9559646Z
Status: 400 (The specified blob or block content is invalid.)

If i remove the InitialTransferSize from the StorageTransferOptions, i get the following error after some time:

retry failed after 6 tries. (The operation was canceled.)

As far as I understood the new SDK, the upload in chunks and therefore the whole handling of the blockIds etc. should be done by the SDK. Or am I wrong?

Does anybody know why this is not working? I did not find anything different then this for BlobContainerClient, only for the old cloudblobcontainer.

Update: Some Additional Informations:

It is a .netCore 3.1 Application which runs with the library Topshelf as a Windows Service

what is this? `await using var uploadFileStream = File.OpenRead(fileInfo.FullName);` Is that some new thing in C# that I am not aware of? — Andy, Sep 10 '20 at 01:21
I ran into this "Retry failed after 6 tries. A task was canceled." trying to export a 3GB Azure SQL database into Azure storage. Worked for many years w/ no problems until this month. — tofutim, Jan 21 '21 at 18:52

score 1 · Answer 1 · answered Feb 02 '21 at 10:19

The second part of your question after you remove the InitialTransferSize from the StorageTransferOptions is similar to the issue in this question.

You may be able to resolve the issue by setting the timeouts for the blob client as follows:

var blobClientOptions = new BlobClientOptions
{
    Transport = new HttpClientTransport(new HttpClient { Timeout = Timeout.InfiniteTimeSpan }),
    Retry = { NetworkTimeout = Timeout.InfiniteTimeSpan }
};

InfiniteTimeSpan is probably overkill, but at least it will prove if that was the issue.

Those settings got rid of the "retry failed after 6 tries" error for me and got the upload when I started using the Azure.Storage.Blobs v12.8.0 package

score 0 · Answer 2 · answered Sep 10 '20 at 03:09

I create a new console app and test with your code which works very well.

1.Confirm that you do not have inconsistencies in assemblies. Remove the earlier version of Azure.Storage.Blobs and update you itto the latest version.

And why your containerClient is private? You could set it in Upload method with following code:

BlobServiceClient blobServiceClient = new BlobServiceClient(connectionstring);
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient("containerName");
var blobClient = containerClient.GetBlobClient(remotePath + "/" +fileInfo.Name);

I have the latest Version (12.6.0) of Azure.Storage.Blobs installed, I also cleaned and reinstalled everything again. Still the same error The containerClient is initialized in the constructor, so that I do not need to initialize it in every method that needs it. — metabolic, Sep 11 '20 at 15:02

score 0 · Accepted Answer · answered Sep 12 '20 at 09:41

0

I was not able to get it working with version 12.6.0...

I downgraded to Microsoft.Azure.Storage.Blob v11 and implemented the upload based on this thread https://stackoverflow.com/a/58741171/765766

This works fine for me now

answered Sep 12 '20 at 09:41

metabolic

669
1
7
24

Large Fileupload to Azure with Blob Storage with BlobContainerClient

3 Answers3