IIS Buffering Requests Content Before Controller Action

Question

I am trying to setup an "external" web api that will be able to receive in large http posts (+1GB) and forward the stream to another "internal" web api that writes the request contents to a file. I have model my implementation based on examples of using a custom WebHostBufferPolicySelector and using the UseBufferedInputStream method in the controller method. It works as expected when using IIS Express, no significate increase in memory footprint, but as soon as my code is deployed to IIS, the memory footprint is substantial and results in OOM.

I have put tracing statements in my controller methods and in my WebHostBufferPolicySelector.UseBufferedInputStream, and have verified that UseBufferedInputStream always is returning false and that my controller methods are getting hit. The only difference that I noticed is that when I debug, the time stamps between UseBufferedInputStream and my controller method are very close. Where hosted on IIS, the time stamps are very far apart, suggesting that something in between when UseBufferedInputStream is called and my controller method is called is buffering up the request entirely.

I am looking for some tips on to find out what that is causing the request to get buffered and how for it not to buffer and using streaming all the way.

Client is coming at the external web api with a content type of application/octet-stream with Transfer Encoding of Chucked.

Used to build out implementation

https://forums.asp.net/t/2018289.aspx?Web+API2+WebHostBufferPolicySelector+UseBufferedInputStream+override

https://www.strathweb.com/2012/09/dealing-with-large-files-in-asp-net-web-api/

Proxy Web Api Controller Method

        <HttpPost, Route("postLargeFile")>
        Protected Overridable Async Function PostLargeFile() As Threading.Tasks.Task(Of IHttpActionResult)
            Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyProxyController)}", "Started {0}", NameOf(MyProxyController.PostLargeFile))

            Dim internalHttpClient As HttpClient
            Dim fowardingContent As StreamContent = Nothing
            Dim fowardingMessage As HttpRequestMessage = Nothing
            Dim fowardingResponse As HttpResponseMessage = Nothing
            Dim externalResponse As HttpResponseMessage = Nothing

            Try
                internalHttpClient = New HttpClient()
                internalHttpClient.BaseAddress = "https://myinternalService.com"

                fowardingMessage = New HttpRequestMessage(HttpMethod.Post, "https://myinternalService.com/saveLargeFile")
                fowardingContent = New StreamContent(HttpContext.Current.Request.GetBufferlessInputStream(True))
                CopyContentHeaders(Request.Content, fowardingContent)

                fowardingMessage.Headers.TransferEncodingChunked = True
                fowardingMessage.Content = fowardingContent

                fowardingResponse = Await internalHttpClient.SendAsync(fowardingMessage, HttpCompletionOption.ResponseHeadersRead)

                externalResponse = New HttpResponseMessage(fowardingResponse.StatusCode)
                externalResponse.Content = New StreamContent(Await fowardingResponse.Content.ReadAsStreamAsync)
                CopyContentHeaders(fowardingResponse.Content, externalResponse.Content)

                Return New Results.ResponseMessageResult(externalResponse)

            Catch ex As Exception
                Return InternalServerError(ex)
            Finally
                Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyProxyController)}", "Finished {0}", NameOf(MyProxyController.PostLargeFile))
            End Try
        End Function

Internal Web Api Controller Method

        <HttpPost, Route("saveLargeFile")>
        Protected Overridable Async Function SaveLargeFile() As Threading.Tasks.Task(Of IHttpActionResult)
            Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyInternalController)}", "Started {0}", NameOf(MyInternalController.PostLargeFile))

            Dim bufferlessStream As IO.Stream
            Dim fowardingContent As StreamContent = Nothing

            Try

                bufferlessStream = HttpContext.Current.Request.GetBufferlessInputStream()
                Using fileStream As IO.FileStream = IO.File.Create("MyFile.txt")
                    bufferlessStream.CopyTo(fileStream)
                    fileStream.Flush()
                End Using

                Return New Results.StatusCodeResult(Net.HttpStatusCode.Created, Me)

            Catch ex As Exception
                Return InternalServerError(ex)
            Finally
                Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyInternalController)}", "Finished {0}", NameOf(MyInternalController.PostLargeFile))
            End Try
        End Function

Policy Selector Configuration

Public Class MyBufferPolicySelector
    Inherits Http.WebHost.WebHostBufferPolicySelector

    Public Property Tracer As ITraceWriter

    Public Overrides Function UseBufferedInputStream(hostContext As Object) As Boolean
        UseBufferedInputStream = False
        Tracer?.Info(Nothing, $"{Me.GetType.Namespace}.{NameOf(MyBufferPolicySelector)}", "{0} UseBufferedInputStream={1}", HttpContext.Current?.Request?.Url?.AbsoluteUri, UseBufferedInputStream)
        Return UseBufferedInputStream
    End Function
End Class

WebApiConfig for both Internal and External Web APIs

Public Module WebApiConfig

    Public Sub Register(ByVal config As HttpConfiguration)
        Dim tracer As SystemDiagnosticsTraceWriter

        ' Web API configuration and services

        ' Web API routes
        config.MapHttpAttributeRoutes()

        tracer = config.EnableSystemDiagnosticsTracing
        tracer.IsVerbose = True
        tracer.MinimumLevel = Tracing.TraceLevel.Debug

        GlobalConfiguration.Configuration.Services.Replace(GetType(IHostBufferPolicySelector), New MyBufferPolicySelector() With {.Tracer = tracer})

    End Sub

End Module

score 0 · Answer 1 · answered May 30 '19 at 18:32

I was able to figure out what was causing the the buffering in IIS. The below link lead me to the uploadReadAheadSize setting in IIS. This was maxed out. So this would cause IIS to fully read in/buffer in the request before passing it into the module where the web api pipeline exists (web api controllers). After setting it to the default, I saw my large file posts not get buffered, the app pool memory footprint remained low, no more out of memory exceptions, and a large performance boost. Great!

But now I have the same issue as described in the below link. When SSL is required, set in IIS, which it is required in our non development environments, the uploadReadAheadSize needs to be increased so the ssl can work in the ssl module I guess. It might have to do with some SSL renegotiation.

Can anybody describe a way to prevent the buffering in SSL to keep the memory footprint low and prevent out of memory exceptions for large http posts?

Large file upload when using ssl and client certificates (uploadReadAheadSize) but dont want all data to be readahead

IIS Buffering Requests Content Before Controller Action

1 Answers1