How can I monitor/log file upload progress on a Kestrel web server?

Question

I am trying to monitor the current file's upload progress server side. I can't find a way to intercept the upload event to log the upload progress of a file. I am uploading large files, so this feature is needed to monitor the requests. It seems that my method (below) isn't called until the file has finished uploading entirely.

Here is a snippet of my method that handles the file upload AFTER it is finished:

[HttpPost("file-upload")]
public async Task<IActionResult> HandleFileUpload(IFormFile file, string? uuid)
{
   ...
}

After an hour of googling, I haven't been able to find any information on this topic. Any ideas on this topic would be very helpful.

Answer 1

First, create a singleton service to hold the upload ID and its progress. Then, whenever you call the main upload API, update its progress. This way, you can retrieve the progress of each upload by its ID using the service.

Take a look at this sample code...

public interface IUploadProgressProvider
{
    void UpdateProgress(string id, double progress);
    double? GetProgress(string id);
    void Remove(string id);
    IEnumerable<(string Id, double Progress)> Info();
}
public sealed class UploadProgressProvider : IUploadProgressProvider
{
    private ConcurrentDictionary<string, double> _uploadProgresses = new(StringComparer.InvariantCultureIgnoreCase);

    public double? GetProgress(string id) => _uploadProgresses.TryGetValue(id, out var r) ? r : null;

    public void Remove(string id) => _uploadProgresses.TryRemove(id, out var r);

    public void UpdateProgress(string id, double progress) => _uploadProgresses.AddOrUpdate(id, progress, (id, p) => progress);

    public IEnumerable<(string Id, double Progress)> Info() => this._uploadProgresses.Select(x => (x.Key, x.Value)).ToList();
}

Route("api/Upload")]
public sealed class UploadController : Controller
{
    private readonly IUploadProgressProvider _uploadProgressProvider;

    public UploadController(IUploadProgressProvider uploadProgressProvider)
    {
        _uploadProgressProvider = uploadProgressProvider;
    }
    [HttpPost("Upload/{id}")]
    public async Task<IActionResult> Index(IList<IFormFile> files, string id)
    {
        long totalBytes = files.Sum(f => f.Length);

        foreach (IFormFile source in files)
        {
            string filename = ContentDispositionHeaderValue.Parse(source.ContentDisposition).FileName.Trim('"');

            byte[] buffer = new byte[16 * 1024];

            using (FileStream output = System.IO.File.Create(this.GetPathAndFilename(filename)))
            {
                using (Stream input = source.OpenReadStream())
                {
                    long totalReadBytes = 0;
                    int readBytes;

                    while ((readBytes = input.Read(buffer, 0, buffer.Length)) > 0)
                    {
                        await output.WriteAsync(buffer, 0, readBytes);
                        totalReadBytes += readBytes;
                        _uploadProgressProvider.UpdateProgress(id, ((float)totalReadBytes / (float)totalBytes * 100.0));
                        await Task.Delay(100); // It is only to make the process slower
                    }
                }
            }
        }
        _uploadProgressProvider.UpdateProgress(id, 100);
        return this.Content("success");
    }
    [HttpGet("info")]
    public async Task<IActionResult> Info() => this.Ok(this._uploadProgressProvider.Info());
    
}

Answer 2

I think it's better to use a combination of server-side code and JS, first of all on client-side,send the file to the server by XMLHttpRequest via JS like this :

const fileInput = document.getElementById('file-input');
const progressBar = document.getElementById('progress-bar');

fileInput.addEventListener('change', (event) => {
  const file = event.target.files[0];
  const formData = new FormData();
  formData.append('file', file);

  const xhr = new XMLHttpRequest();
  xhr.upload.addEventListener('progress', (event) => {
    const progress = Math.round((event.loaded / event.total) * 100);
    progressBar.style.width = `${progress}%`;
  });

  xhr.open('POST', '/file-upload');
  xhr.send(formData);
});

now in server-side,edit your HandleFileUpload method to accept a stream instead of the entire file and by reading the file in chunks and logging the progress, you are able to monitor the upload progress on the server-side!

[HttpPost("file-upload")]
public async Task<IActionResult> HandleFileUpload(string? uuid)
{
    var stream = Request.Body;
    //you can change chunk size as your requirements
    var buffer = new byte[8192]; // chunk_size

    long totalBytesRead = 0;
    int bytesRead;

    while ((bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length)) > 0)
    {
        //processing the chunk and updating the progress based on the bytes read
        totalBytesRead += bytesRead;

        double progress = (double)totalBytesRead / Request.ContentLength * 100;
        Console.WriteLine($"Upload progress: {progress}%");
    }

    //upload completed
    return Ok();
}