C# Azure AppendBlob AppendBlock adding a file larger than the 4mb limit

Question

I've searched and searched and have not found any examples.

I'm using the Azure.Storage.Blobs nuget packages in C# .NET Core.

Here is an example of my current code that doesn't work.

I get a Status: 413 (The request body is to large and exceeds the maximum permissible limit.)

Searching seems to indicate there is either a 4mb limit or a 100mb limit it's not clear but I think it's 4mb on Append Blobs and 100mb limit on Block Blobs.

var appendBlobClient = containerClient.GetAppendBlobClient(string.Format("{0}/{1}", tenantName, Path.GetFileName(filePath)));

using FileStream uploadFileStream = File.OpenRead(filePath);
appendBlobClient.CreateIfNotExists();
appendBlobClient.AppendBlock(uploadFileStream);
uploadFileStream.Close();

This doesn't work because of the 4mb limit so I need to append 4mb chunks of my file but I've not found examples of the best way to do this.

So what I'm trying to figure out is the best way to upload large files it seems it has to be done in chunks maybe 4mb for append blobs and 100mb for block blobs but the documentation isn't clear and doesn't have examples.

Saying "it does not work" is not a helpful start. Please add error messages etc. Also, have you looked into the API docs? https://learn.microsoft.com/en-us/dotnet/api/azure.storage.blobs.specialized.appendblobclient.appendblockasync?view=azure-dotnet#Azure_Storage_Blobs_Specialized_AppendBlobClient_AppendBlockAsync_System_IO_Stream_System_Byte___Azure_Storage_Blobs_Models_AppendBlobRequestConditions_System_IProgress_System_Int64__System_Threading_CancellationToken_ — silent, Feb 19 '20 at 17:16
Sorry about that I've edited my question to include more detail. — Chris Ward, Feb 19 '20 at 17:30
Just combine the `AppendBlock()` with something like this? https://stackoverflow.com/a/6865956/1537195 — silent, Feb 19 '20 at 17:44
Just to double check: Why are you using Append Blobs vs Block Blobs? — silent, Feb 19 '20 at 17:45
@silent The first upload will be a large amount of data after that only deltas will be appended to it. So Append Blob makes sense for this. I did just test a blockblob and was able to upload a 6gb file and from I'm reading block blob upload method handels the chunking for you and appears the AppendBlob AppendBlock method does not so for appends I'll have to make my own chunks — Chris Ward, Feb 19 '20 at 18:03

score 5 · Answer 1 · edited May 05 '20 at 20:05

I want to thank @silent for responding since he provided enough info to work out what I needed. Sometimes just having someone to talk through things helps me figure things out.

What I found in on the BlockBlobClient.Upload method it chunks your file stream for you. I believe this to be 100mb blocks from my research. It appears it has a limit of 100mb blocks and 50,000 of them

For AppendBlockClient.AppendBlock it does not chunk your stream for you. It has a limit of 4mb blocks and 50,000 of them.

Here is part of my code that allowed me to upload a 6gb file as a block blob and a 200mb file as an append blob.

BlobServiceClient blobServiceClient = new BlobServiceClient(azureStorageAccountConnectionString);
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(azureStorageAccountContainerName);
containerClient.CreateIfNotExists();

if (appendData)
{
    var appendBlobClient = containerClient.GetAppendBlobClient(string.Format("{0}/{1}", tenantName, Path.GetFileName(filePath)));

    appendBlobClient.CreateIfNotExists();

    var appendBlobMaxAppendBlockBytes = appendBlobClient.AppendBlobMaxAppendBlockBytes;
    using (var file = File.OpenRead(filePath))
    {
        int bytesRead;
        var buffer = new byte[appendBlobMaxAppendBlockBytes];
        while ((bytesRead = file.Read(buffer, 0, buffer.Length)) > 0)
        {
            //Stream stream = new MemoryStream(buffer);
            var newArray = new Span<byte>(buffer, 0, bytesRead).ToArray();
            Stream stream = new MemoryStream(newArray);
            stream.Position = 0;
            appendBlobClient.AppendBlock(stream);
        }
    }
}
else
{
    var blockBlobClient = containerClient.GetBlockBlobClient(string.Format("{0}/{1}", tenantName, Path.GetFileName(filePath)));

    using FileStream uploadFileStream = File.OpenRead(filePath);
    blockBlobClient.DeleteIfExists();
    blockBlobClient.Upload(uploadFileStream);
    uploadFileStream.Close();
}

glad it worked. Btw, I would highly recommend to always use the Async versions of the APIs, such as `AppendBlockAsync()` — silent, Feb 20 '20 at 09:21
@silent can you tell me the reasons why? I started to do this and being this is a .NET Core Console app I ran into all kinds of problems and a lot of extra work to be able to call async methods that required awaits and then tasks to return values from methods. — Chris Ward, Feb 20 '20 at 13:22
In a nutshell: Basically always use async! https://stackoverflow.com/a/48023725/1537195 Btw, in a Console app, you can start with a `public static async Task Main(string[] args)` to make your life a whole lot easier. From there on its not much difference to program — silent, Feb 20 '20 at 13:27
Thanks, got it all sorted out now and changed it all over to async where those methods were available. — Chris Ward, Feb 20 '20 at 14:56
getting Status: 409 (The blob type is invalid for this operation.) on line appendBlobClient.AppendBlock(stream); — winterishere, Jun 27 '21 at 17:21

C# Azure AppendBlob AppendBlock adding a file larger than the 4mb limit

1 Answers1

Linked