ConnectionResetException: An existing connection was forcibly closed by the remote host on Rest API call for big file

Question

I have an C# WebAPI that let user upload file in chunks. The application works fine for http. But when I added https for my API I found out it works fine for small size of files, but for bigger file sizes (>60MB) the request just waits for 2 minutes and then just fails. After looking at the server log I found this exception:

Microsoft.AspNetCore.Connections.ConnectionResetException: An existing connection was forcibly closed by the remote host.
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.<GetResult>g__ThrowSocketException|5_0(SocketError e)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.GetResult(Int16 token)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.DoReceive()
   --- End of inner exception stack trace ---
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.IO.Pipelines.PipeReader.CopyToAsyncCore[TStream](TStream destination, Func`4 writeAsync, CancellationToken cancellationToken)
   at Infrastructure.LocalStorage.Domain.UploadFile.UploadChunkFileStreamService.SaveFileChunkAsync(UploadFileCommand command, String fileChunkPath)

What's even more crazy is it works fine for one server (we set up https on dev environment) but not for the other server. In both cases we are using IIS. Any idea how we can fix this issue? Am I missing any IIS configuration for HTTPS?

Note:

• we already made uploadReadAheadSize bigger to avoid 413 error.
• added the following code to my server as suggested on this post (however that's a SocketException NOT ConnectionResetException although the message are same)

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 

• the file is being uploaded by a react app from the front-end.

Note 2: When I lower the number of chunks to upload to the server it works fine without any error. For now I am uploading 2 chunks at a time.

Answer 1

If you are posting a lot of data, you need to specify the request limit, length and execution maximum time in Web.config. The answer comes from this thread.

<requestLimits maxAllowedContentLength="2147483647" />
<httpRuntime targetFramework="4.6.1" maxRequestLength="2147483647" executionTimeout="1600" requestLengthDiskThreshold="2147483647" />

In addition, this explains the setting in detail.

Answer 2

For a IIS hosted application it's important set on web.config requestLimits tag for more information : https://learn.microsoft.com/en-us/iis/configuration/system.webserver/security/requestfiltering/requestlimits/ and in stackoverflow (already answered) maxRequestLength for .Net 4.5.1 framework

also increase requesttimeout on web.config file:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified"/>
    </handlers>
    <aspNetCore requestTimeout="00:15:00"  processPath="%LAUNCHER_PATH%" arguments="%LAUNCHER_ARGS%" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" />
  </system.webServer>
</configuration>

Answer 3

There's two parts to this:

1. Configuring the .net WebAPI back-end application. The answers above cover the app.config settings as well as the C# calls needed.
2. Configuring the web server to accept files larger than X megabytes

There are also, unlikely, a part 3, 4, 5 or 6 sometimes

3. If you are using cloud based Azure web app, it requires a premium web app service plan. The basic web app service plan limits files to 60 megabytes.

4. Any firewall, proxy, etc. in-between your front-end caller code and the back-end server could drop the request due to size, or timeout

5. If you are uploading multiple file chunks at the same time, the web server could run out of resources and block.

6. Some sort of deadlock, file write blocking, or out of memory on the C# back-end server due to the amount of data. I'd explicitly clear any C# byte array, close any file, close and dispose of any stream, etc and set the variables to null. The C# garbage collector will do this eventually, but it is better to force them to be cleaned up.

Answer 4

"SocketException (10054): An existing connection was forcibly closed by the remote host" means that a client disconnected ungracefully. This can happen if the client process exited mid-connection for example.

If the client disconnects in the middle of uploading a request body, that can cause HttpContext.Request.Body.ReadAsync() to throw leading to higher severity logs, which is pretty rare.

If you just want to fix, you could try enabling buffering and drain the entire request body in a try/catch. In the catch, you can log the issue with whatever severity you want. If you successfully read the full body, you can set request.Body.Position back to zero and read again from your main app without any concern over seeing an IOException.

The suggested way to enable request body for multiple reads. Here is an example usage in the InvokeAsync() method of a custom ASP.NET middleware:

public async Task InvokeAsync(HttpContext context, RequestDelegate next)
{
context.Request.EnableBuffering();

// Leave the body open so the next middleware can read it.
using (var reader = new StreamReader(
    context.Request.Body,
    encoding: Encoding.UTF8,
    detectEncodingFromByteOrderMarks: false,
    bufferSize: bufferSize,
    leaveOpen: true))
    {
    var body = await reader.ReadToEndAsync();
    // Do some processing with body…

    // Reset the request body stream position so the next middleware can read it
    context.Request.Body.Position = 0;
    }

    // Call the next delegate/middleware in the pipeline
    await next(context);
    }

The backing FileBufferingReadStream uses memory stream of a certain size first then falls back to a temporary file stream. By default the size of the memory stream is 30KB. There are also other EnableBuffering() overloads that allow specifying a different threshold, and/or a limit for the total size:

public static void EnableBuffering(this HttpRequest request, int bufferThreshold)
public static void EnableBuffering(this HttpRequest request, long bufferLimit)
public static void EnableBuffering(this HttpRequest request, int bufferThreshold, long bufferLimit)

I hope this will be helpful for resolving the issue.

Answer 5

I also had the error I fixed it after a long tests.

Go to Windows Features > uncheck Internetinformationservices

an now it work's for me on Debug .Net6 / .Net7 Blazor Server App