Calculating Time Remaining on File Copy

Question

I have an app that copies a large amount of files across the network to a file server (not web). I am trying to display a half decent estimation of the time remaining.

I have looked at a number of articles on SO an while the problem is addressed none that I have tried really do what I want. I want the estimated time remaining to be relatively stable I.E. not jump around all over the place depending on fluctuating transfer speeds.

So the first solution I looked at was to calculate the transfer speed in bytes per second

double bytePerSec = totalBytesCopied / TimeTaken.TotalSeconds;

And then divide the total byte remaining by the transfer rate.

double secRemain = (totalFileSizeToCopy - totalBytesCopied) / bytePerSec;

I figured that the time remaining would become more stable once a few MB had been copied (although expecting it to change . It doesn't, its erratic and jumps around all over the place.

Then I tried one of the solutions on SO....

double secRemain = (TimeTaken.TotalSeconds / totalBytesCopied) * (totalFileSizeToCopy - totalBytesCopied);

Which is a similar calculation but hoped it might make a difference!

So now I am kind of thinking I need to approach this from a different angle. IE Use averages? Use some kind of countdown timer and reset the time to go every so often? Just looking for opinions or preferably advice from anyone that has already had this problem.

Answer 1

Here's a working example of how you would asynchronously copy a file D:\dummy.bin to D:\dummy.bin.copy, with a timer taking snapshots of the transfer rate every second.

From that data, I simply take the average transfer rate from up to 30 snapshots (newest first). From that I can calculate a rough estimate of how long it will take to transfer the rest of the file.

This example is provided as-is and does not support copying multiple files in 1 operation. But it should give you some ideas.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;

public class Program
{
    public static void Main(string[] args)
    {
        var sourcePath = @"D:\dummy.bin";
        var destinationPath = @"D:\dummy.bin.copy";
        var sourceFile = new FileInfo(sourcePath);
        var fileSize = sourceFile.Length;
        var currentBytesTransferred = 0L;
        var totalBytesTransferred = 0L;
        var snapshots = new Queue<long>(30);
        var timer = new System.Timers.Timer(1000D);
        timer.Elapsed += (sender, e) =>
        {
            // Remember only the last 30 snapshots; discard older snapshots
            if (snapshots.Count == 30)
            {
                snapshots.Dequeue();
            }

            snapshots.Enqueue(Interlocked.Exchange(ref currentBytesTransferred, 0L));
            var averageSpeed = snapshots.Average();
            var bytesLeft = fileSize - totalBytesTransferred;
            Console.WriteLine("Average speed: {0:#} MBytes / second", averageSpeed / (1024 * 1024));
            if (averageSpeed > 0)
            {
                var timeLeft = TimeSpan.FromSeconds(bytesLeft / averageSpeed);
                var timeLeftRounded = TimeSpan.FromSeconds(Math.Round(timeLeft.TotalSeconds));
                Console.WriteLine("Time left: {0}", timeLeftRounded);
            }
            else
            {
                Console.WriteLine("Time left: Infinite");
            }
        };

        using (var inputStream = sourceFile.OpenRead())
        using (var outputStream = File.OpenWrite(destinationPath))
        {
            timer.Start();
            var buffer = new byte[4096];
            var numBytes = default(int);
            var numBytesMax = buffer.Length;
            var timeout = TimeSpan.FromMinutes(10D);
            do
            {
                var mre = new ManualResetEvent(false);
                inputStream.BeginRead(buffer, 0, numBytesMax, asyncReadResult =>
                {
                    numBytes = inputStream.EndRead(asyncReadResult);
                    outputStream.BeginWrite(buffer, 0, numBytes, asyncWriteResult =>
                    {
                        outputStream.EndWrite(asyncWriteResult);
                        currentBytesTransferred = Interlocked.Add(ref currentBytesTransferred, numBytes);
                        totalBytesTransferred = Interlocked.Add(ref totalBytesTransferred, numBytes);
                        mre.Set();
                    }, null);
                }, null);
                mre.WaitOne(timeout);
            } while (numBytes != 0);
            timer.Stop();
        }
    }
}

Answer 2

You want to calculate the rate based on average transfer rate, but you want it to be a moving average since network speed is variable over the the life of the file transfer (especially for very large files). Here's the JavaScript method I came up with that seems to work well (should be easy to convert to C#).

var rates = [];
var ratesLength = 1000;
for (var i = 0; i < ratesLength; i++)
    rates[i] = 0;

/**
* Estimates the remaining download time of the file.
*
* @param {number} bytesTransferred: Number of bytes transferred so far.
* @param {number} totalBytes: Total number of bytes to be transferred.
* @return {string} Returns the estimating time remaining in the upload/download.
*/
function getSpeed(bytesTransferred, totalBytes) {

    var bytes = bytesTransferred - oldBytesTransfered;
    var time = currentTime - oldTime;

    if ((time != 0) && (bytes != 0)) {
        rates[rateIndex] = (bytes) / (time);
        rateIndex = (rateIndex + 1) % rates.length;
        var avgSpeed = 0;
        var count = 0;
        for (i = 0; i < rates.length ; i++) {
            if (rates[i] != 0) {
                avgSpeed += rates[i];
                count++;
            }
        }
        if (count == 0)
            return " ";

        avgSpeed /= count;
        return (humanReadableTime((totalBytes - bytesTransferred) / avgSpeed) + " remaining");
    } else {
        return " ";
    }
}

You can tweak the ratesLength to get the smoothness you want. oldTime is the time that the last chuck of bytes were received and oldBytesTransfered is the number of total bytes transferred after the last chunk. bytesTransferred is the total amount transferred including the current chuck.

I've found that doing a weighted average like this has good accuracy, but also reacts well to changes in speed.

Answer 3

You'll likely want to calculate the estimate against the average transfer rate. That should help stabilize the estimate. There's another SO question which asks the same question.

Answer 4

This could be one aproach

Calculate the bytes/second when the 25% of the first file has been copied

Use that calculation to estimate the rest of the 75% of the first file operation

Every time that a file has been copied store, in a collection, the average of the speed in bytes/second for each copied file.

Then you can use the average of the items in the collection to estimate the time of the of the copy operation for the files that have not yet been copied.

Let me know if you need more details or aproaches