Way to limit azure blob upload speed

Question

I have an application that uploads database backups to a blob container on azure.

I'm uploading files around 8GB in size, at an average rate of 11Mbps

Everything is working fine however, whilst the blob is being transferred to the container, it absolutely kills everything else on that network.

So my question is, is there any way to limit the upload speed to azure?

Answer 1

Your question didn't specify in which programming language (or raw REST) you would want to achieve the upload speed throttling. But I found a way to do this using the official azure storage .NET SDK (Microsoft.WindowsAzure.Storage).

The SDK itself doesn't provide explicit speed throttling. However, the Upload*Async() functions, e.g., UploadFromFileAsync() does support progress reporting, by taking a IProgress<StorageProgress> implementation, and call its Report() method periodically. And it appears that it calls it synchronously.

So we can put a delay in Report() to throttle the upload speed. What's even better, is that the Report gives you info on BytesTransferred. So if your progress handler keeps track of duration. Then you can estimate the current upload speed, and use that to throttle precisely.

Below is an illustration of this idea. Note that this is merely a demonstration of the idea. I put it together in less than 3 minutes. The rate estimation and throttling algorithm is quite coarse and not well tested.

Here's how you would use it

await blob.UploadFromFileAsync(
    @"some_file.dat", 
    null, null, null, 
    new RateThrottleProgress(300 * 1024), // throttle at 300kb/s
    CancellationToken.None);

Exponential back-off Throttler implementation

class RateThrottleProgress : IProgress<StorageProgress>
{
    private readonly DateTime start = DateTime.Now;
    private readonly long maxbps;
    private long baseDelay, delay;

    public RateThrottleProgress(long maxbps)
    {
        this.maxbps = maxbps;
        baseDelay = 10;
        delay = baseDelay;
    }

    public void Report(StorageProgress value)
    {
        double duration = (DateTime.Now - start).TotalSeconds;
        double bps = value.BytesTransferred / duration;
        if (bps > maxbps) delay *= 2;
        else delay = Math.Max(baseDelay, delay/2);
        Console.WriteLine($"current estimated upload speed: {bps / 1024.0} KB/s. delay: {delay} ms");
        Thread.Sleep(TimeSpan.FromMilliseconds(delay));
    }
}

I also put below code snippet as a gist for better version control and collaboration.

Throttle in effect:

current estimated upload speed: 287.486007463505 KB/s. delay: 10 ms
current estimated upload speed: 290.086402388889 KB/s. delay: 10 ms
current estimated upload speed: 292.685419108659 KB/s. delay: 10 ms
current estimated upload speed: 295.28201245662 KB/s. delay: 10 ms
current estimated upload speed: 297.876060423937 KB/s. delay: 10 ms
current estimated upload speed: 300.469027029562 KB/s. delay: 20 ms
current estimated upload speed: 302.927815243916 KB/s. delay: 40 ms
current estimated upload speed: 305.112558483135 KB/s. delay: 80 ms
current estimated upload speed: 306.778888691779 KB/s. delay: 160 ms
current estimated upload speed: 307.367196107083 KB/s. delay: 320 ms   <-- speed starts to drop from here...
current estimated upload speed: 305.910611140488 KB/s. delay: 640 ms
current estimated upload speed: 300.564767027164 KB/s. delay: 1280 ms
current estimated upload speed: 288.206861583389 KB/s. delay: 640 ms
current estimated upload speed: 283.672713628354 KB/s. delay: 320 ms
current estimated upload speed: 282.668039190231 KB/s. delay: 160 ms
current estimated upload speed: 283.351226090087 KB/s. delay: 80 ms
current estimated upload speed: 284.861107569046 KB/s. delay: 40 ms
current estimated upload speed: 286.781960850501 KB/s. delay: 20 ms
current estimated upload speed: 288.910675693183 KB/s. delay: 10 ms
current estimated upload speed: 291.140146046991 KB/s. delay: 10 ms
current estimated upload speed: 293.358817316007 KB/s. delay: 10 ms

Answer 2

Azure Storage itself won't provide a specific throttle setting. However: Given that a blob upload is essentially a series of REST-based block blob PUT operations, you could do your own throttling by doing your own upload implementation and choosing how frequently to upload blocks to a given blob.

Answer 3

It might be a better approach to setup a maximum upload bandwith on the network interface. This is completely transparent to your client application and a good design goal. Your software in another QoS network will work without custom throttling.

This SO question is about limiting network speed and lists some tools for this purpose. If the machine running database backups is using Windows try something like BWMeter to limit network speed.

A much better strategy would be to configure the network such that the Internet gateway (WAN/DSL router) is doing traffic shaping. It depends on the product if that works well enough.

Both approaches should have lesser long-terms cost than the alternative way that implements a custom throttling into your application.