Design Minimal API and use HttpClient to post a file to it

Question

I have a legacy system interfacing issue that my team has elected to solve by standing up a .NET 7 Minimal API which needs to accept a file upload. It should work for small and large files (let's say at least 500 MiB). The API will be called from a legacy system using HttpClient in a .NET Framework 4.7.1 app.

I can't quite seem to figure out how to design the signature of the Minimal API and how to call it with HttpClient in a way that totally works. It's something I've been hacking at on and off for several days, and haven't documented all of my approaches, but suffice it to say there have been varying results involving, among other things:

• 4XX and 500 errors returned by the HTTP call
• An assortment of exceptions on either side
• Calls that throw and never hit a breakpoint on the API side
• Calls that get through but the Stream on the API end is not what I expect
• Errors being different depending on whether the file being uploaded is small or large
• Text files being persisted on the server that contain some of the HTTP headers in addition to their original contents

On the Minimal API side, I've tried all sorts of things in the signature (IFormFile, Stream, PipeReader, HttpRequest). On the calling side, I've tried several approaches (messing with headers, using the Flurl library, various content encodings and MIME types, multipart, etc).

This seems like it should be dead simple, so I'm trying to wipe the slate clean here, start with an example of something that partially works, and hope someone might be able to illuminate the path forward for me.

Example of Minimal API:

// IDocumentStorageManager is an injected dependency that takes an int and a Stream and returns a string of the newly uploaded file's URI

app.MapPost(
    "DocumentStorage/CreateDocument2/{documentId:int}",
    async (PipeReader pipeReader, int documentId, IDocumentStorageManager documentStorageManager) =>
    {
        using var ms = new MemoryStream();
        await pipeReader.CopyToAsync(ms);
        ms.Position = 0;
        return await documentStorageManager.CreateDocument(documentId, ms);
    });

Call the Minimal API using HttpClient:

    // filePath is the path on local disk, uri is the Minimal API's URI

    private static async Task<string> UploadWithHttpClient2(string filePath, string uri)
    {
        var fileStream = File.Open(filePath, FileMode.Open);
        var content = new StreamContent(fileStream);
        var httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, uri);
        var httpClient = new HttpClient();
        
        httpRequestMessage.Content = content;
        httpClient.Timeout = TimeSpan.FromMinutes(5);

        var result = await httpClient.SendAsync(httpRequestMessage);

        return await result.Content.ReadAsStringAsync();
    }

In the particular example above, a small (6 bytes) .txt file is uploaded without issue. However, a large (619 MiB) .tif file runs into problems on the call to httpClient.SendAsync which results in the following set of nested Exceptions:

System.Net.Http.HttpRequestException - "Error while copying content to a stream."
  System.IO.IOException - "Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.."
    System.Net.Sockets.SocketException - "An existing connection was forcibly closed by the remote host."
    

What's a decent way of writing a Minimal API and calling it with HttpClient that will work for small and large files?

Answer 1

Kestrel allows uploading 30MB per default.
To upload larger files via kestrel you might need to increase the max size limit. This can be done by adding the "RequestSizeLimit" attribute. So for example for 1GB:

app.MapPost(
    "DocumentStorage/CreateDocument2/{documentId:int}",
    [RequestSizeLimit(1_000_000_000)] async (PipeReader pipeReader, int documentId) =>
    {
        using var ms = new MemoryStream();
        await pipeReader.CopyToAsync(ms);
        ms.Position = 0;
        return "";
    });

You can also remove the size limit globally by setting

builder.WebHost.UseKestrel(o => o.Limits.MaxRequestBodySize = null);

Answer 2

This answer is good but the RequestSizeLimit filter doesn't work for minimal APIs, it's an MVC filter. You can use the IHttpMaxRequestBodySizeFeature to limit the size (assuming you're not running on IIS). Also, I made a change to accept the body as a Stream. This avoids the memory stream copy before calling the CreateDocument API:

app.MapPost(
    "DocumentStorage/CreateDocument2/{documentId:int}",
    async (Stream stream, int documentId, IDocumentStorageManager documentStorageManager) =>
    {
        return await documentStorageManager.CreateDocument(documentId, stream);
    })
    .AddEndpointFilter((context, next) =>
    {
        const int MaxBytes = 1024 * 1024 * 1024;

        var maxRequestBodySizeFeature = context.HttpContext.Features.Get<IHttpMaxRequestBodySizeFeature>();

        if (maxRequestBodySizeFeature is not null and { IsReadOnly: true })
        {
            maxRequestBodySizeFeature.MaxRequestBodySize = MaxBytes;
        }

        return next(context);
    });

If you're running on IIS then https://learn.microsoft.com/en-us/iis/configuration/system.webserver/security/requestfiltering/requestlimits/#configuration