HttpClient on WebApi is extremely slow

Question

I'm implementing a Proxy for my application using ASP.NET WebApi (ApiController) and using HttpClient to make the request with my authorization header. It works fine, but it's extremely slow. Below is the main code, then the Global initialization (with DefaultConnectionLimit) and web.config related piece.

As you can see, I'm already using a static/shared HttpClient object with no Proxy and HttpCompletionOption.ResponseHeadersRead on the actual request. This WebApi endpoint is called in parallel, which works fine.

The entire code runs fast enough, but as I'm using ResponseHeadersRead async, the HttpRequestMessage is returned and the rest of the body is downloaded and streamed directly to the client/caller.

Here is a video showing the problem.

public class ProxyController : ApiController
{
  private const string BASE_URL = "https://developer.api.autodesk.com";
  private const string PROXY_ROUTE = "api/viewerproxy/";


  // HttpClient has been designed to be re-used for multiple calls. Even across multiple threads. 
  // https://stackoverflow.com/questions/22560971/what-is-the-overhead-of-creating-a-new-httpclient-per-call-in-a-webapi-client
  private static HttpClient _httpClient;

  [HttpGet]
  [Route(PROXY_ROUTE + "{*.}")]
  public async Task<HttpResponseMessage> Get()
  {
    if (_httpClient == null)
    {
      _httpClient = new HttpClient(new HttpClientHandler() 
         {
            UseProxy = false,
            Proxy = null
         });
      _httpClient.BaseAddress = new Uri(BASE_URL);
    }

    string url = Request.RequestUri.AbsolutePath.Replace(PROXY_ROUTE, string.Empty);
    string absoluteUrl = url + Request.RequestUri.Query;

    try
    {
      HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, absoluteUrl);
      request.Headers.Add("Authorization", "Bearer " + AccessToken);

      HttpResponseMessage response = await _httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);

      return response;
    }
    catch (Exception e)
    {
      return new HttpResponseMessage(System.Net.HttpStatusCode.InternalServerError);
    }
  }
}

Global.asax, although I don't believe is a problem of connection limit as all the requests are processed, but just too slow...

public class Global : System.Web.HttpApplication
{
  protected void Application_Start(object sender, EventArgs e)
  {
    GlobalConfiguration.Configure(Config.WebApiConfig.Register);

    ServicePointManager.UseNagleAlgorithm = true;
    ServicePointManager.Expect100Continue = false;
    ServicePointManager.CheckCertificateRevocationList = true;
    ServicePointManager.DefaultConnectionLimit = int.MaxValue;
  }
}

And part of the Web.Config

 <system.web>
    <compilation debug="true" targetFramework="4.6" />
    <httpRuntime targetFramework="4.6" maxRequestLength="2097151" requestLengthDiskThreshold="16384" requestPathInvalidCharacters="&lt;,&gt;,*,%,&amp;,\,?" />
  </system.web>

Have you tried to profile your application? Which part of it seems slow? — Andrii Litvinov, Jun 21 '17 at 18:07
as I'm using ResponseHeadersRead on the SendAsync call, the entire code runs and the HttpRequestMessage is returned really fast, but then the HttpClient continues downloading directly and sending to the client, and this "stream" is really slow — Augusto Goncalves, Jun 21 '17 at 18:18
Could it be that communication between you proxy and downstream service is low? Another thing what if you try to buffer response (remove `HttpCompletionOption.ResponseHeadersRead`)? Is it slow for single request or or under load? — Andrii Litvinov, Jun 21 '17 at 18:26
I tried to buffer the response and also single calls, sample problem: the HttpClient download is slow — Augusto Goncalves, Jun 21 '17 at 18:28
Have you tried to perform same request (as proxy does) in browser or in postman? Seems like either network connection between servers is slow or server returns content slowly to http client. — Andrii Litvinov, Jun 21 '17 at 18:31
I have added a quick video with the speed, the client is downloading the data really slow — Augusto Goncalves, Jun 21 '17 at 18:36
ok, good point: if I open a single URL on the browser, it's fast... not sure why as the client works fine without the proxy — Augusto Goncalves, Jun 21 '17 at 18:38
Thanks for the video, it seems that CPU is utilized by 100% while processing requests. So maybe CPU is the bottleneck? — Andrii Litvinov, Jun 21 '17 at 18:50
don't think so.. unless I'm doing something really bad. This code is about ~5 request at a time of around 1mb each... — Augusto Goncalves, Jun 21 '17 at 18:52

score 1 · Accepted Answer · answered Jun 21 '17 at 20:46

Solved by removing the <system.diagnostics> section of the web.config. It seems that it was causing an excess of output and slowing down all HttpClient requests.

For the record, this the code I was using and causing the slowness on all HttpClient.SendAsync calls. But this is useful for tracking connection problems :-)

<system.diagnostics>
  <sources>
    <source name="System.Net" tracemode="protocolonly" maxdatasize="1024">
      <listeners>
        <add name="System.Net"/>
      </listeners>
    </source>
    <source name="System.Net.Cache">
      <listeners>
        <add name="System.Net"/>
      </listeners>
    </source>
    <source name="System.Net.Http">
      <listeners>
        <add name="System.Net"/>
      </listeners>
    </source>
  </sources>
  <switches>
    <add name="System.Net" value="Verbose"/>
    <add name="System.Net.Cache" value="Verbose"/>
    <add name="System.Net.Http" value="Verbose"/>
    <add name="System.Net.Sockets" value="Verbose"/>
    <add name="System.Net.WebSockets" value="Verbose"/>
  </switches>
  <sharedListeners>
    <add name="System.Net"
      type="System.Diagnostics.TextWriterTraceListener"
      initializeData="network.log"
    />
  </sharedListeners>
  <trace autoflush="true"/>
</system.diagnostics>

score 0 · Answer 2 · answered Dec 09 '22 at 05:54

0

5 years later, same problem for me came up. I resolved it by removing any Analyzers inside the VS project's property settings:

answered Dec 09 '22 at 05:54

Fandango68

4,461
4
39
74

HttpClient on WebApi is extremely slow

2 Answers2

Linked