File upload not hitting controller before upload is finished and temporarily saved to disk

Question

Let's imagine I am running a .NET Core API on a device with 1GB storage and 1GB RAM. I want to upload a file from my website directly to a FTP server, without the file getting cached in memory or on disk. I have this working for downloading files, as I basically act as a proxy, by opening the FTP file in a stream, then streaming that directly to the HttpContext.Request.Body.

For uploading, I want to hit the controller immediately. I can see it caches to disk now, and that's probably because of my EnableBuffering attribute. I have a regular <form method="post" enctype="multipart/form-data"> that POSTs to my .NET Core backend. The controller looks like this:

[EnableBuffering]
[RequestFormLimits(ValueLengthLimit = int.MaxValue, MultipartBodyLengthLimit = long.MaxValue)]
[HttpPost("[action]")]
public async Task<IActionResult> Upload(string path)
{
    if (!IsMultipartContentType(HttpContext.Request.ContentType))
        return BadRequest("Not multipart request");

    await _fileProvider.Upload(path);

    return Ok();
}

EnableBuffering:

public class EnableBufferingAttribute : Attribute, IResourceFilter
{
    public void OnResourceExecuting(ResourceExecutingContext context)
    {
        context.HttpContext.Request.EnableBuffering();
    }

    public void OnResourceExecuted(ResourceExecutedContext context) {}
}

Upload logic:

public async Task Upload(string path)
{
    const int buffer = 8 * 1024;

    var request = _httpContextAccessor.HttpContext.Request;
    var boundary = request.GetMultipartBoundary();
    var reader = new MultipartReader(boundary, request.Body, buffer);
    var section = await reader.ReadNextSectionAsync();

    using (var client = await _ftpHelper.GetFtpClient())
    {
        while (section != null)
        {
            var fileSection = section.AsFileSection();

            if (fileSection != null)
            {
                var fileName = fileSection.FileName;

                var uploadPath = Path.Combine(path, fileName);

                var stream = await client.OpenWriteAsync(uploadPath);
                await section.Body.CopyToAsync(stream, buffer);
            }

            section = await reader.ReadNextSectionAsync();
        }
    }
}

If I don't enable buffering, I get:

System.IO.IOException: Unexpected end of Stream, the content may have already been read by another component.

However, by enabling buffering, I can see it says:

Ensure the requestBody can be read multiple times. Normally buffers request bodies in memory; writes requests larger than 30K bytes to disk.

So that part makes sense. However, how do I get around this problem? If I set a breakpoint on the first line of the Upload() method, and then start an upload, it uploads the file first AND THEN hits the breakpoint.

Can I get around this so it hits my controller immediately and starts uploading to the FTP server, without saving to disk first?

Try using log output instead of setting a breakpoint, as it may interfere with upload workflow — Hirasawa Yui, Jul 09 '20 at 13:50
@HirasawaYui I can guarantee it does not hit the controller before upload is finished. Tested by building and deploying to a production environment. — MortenMoulder, Jul 09 '20 at 14:00
You will need buffering, but just use a small buffer. The default buffersize used in CopyTo is 82kb, which means that you can transfer huge files while only using 82kb of memory per transfer — Malte R, Jul 09 '20 at 20:30
@MalteR Yep, that is correct, however, that won't solve my issue with files being cached on disk. Even if I remove the EnableBuffering attribute, it still uploads the files first while saving to disk temporarily, and then it hits my controller. — MortenMoulder, Jul 10 '20 at 06:26

score 3 · Answer 1 · answered Feb 23 '22 at 06:10

The reason why request is processed prior entering the Upload method is the Form model binding. Upload method has path parameter and it value comes from the form data. As result, the request need to be parsed prior calling controller method to extract path parameter value from form data. Also in this case, the list and content of uploaded files can be retrieved using Files collection available via HttpContext property of the ControllerBase class.

this.HttpContext.Request.Form.Files

The easiest way to workaround this behavior is to disable form model binding and pass the path parameter via the query string.

[HttpPost("[action]")]
[DisableFormValueModelBinding]
public async Task<IActionResult> Upload([FromQuery] string path)

The source code of DisableFormValueModelBindingAttribute can be found in Upload files in ASP.NET Core article.

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public class DisableFormValueModelBindingAttribute : Attribute, IResourceFilter
{
    public void OnResourceExecuting(ResourceExecutingContext context)
    {
        var factories = context.ValueProviderFactories;
        factories.RemoveType<FormValueProviderFactory>();
        factories.RemoveType<FormFileValueProviderFactory>();
        factories.RemoveType<JQueryFormValueProviderFactory>();
    }

    public void OnResourceExecuted(ResourceExecutedContext context)
    {
    }
}

File upload not hitting controller before upload is finished and temporarily saved to disk

1 Answers1