.net HttpClient in C#: Exception when re-instantiating

Question

Am querying multiple APIs with thousands of requests. Thus, I am looping over the end points and the requests. As it is suggested to re-use HttpClient instances, that's what I am doing. However, I need to set some parameters like timeouts, passwords etc. in the header for each API. Thus, the first API works perfectly, when trying to set the Parameters for the next API, it fails:

This instance has already started one or more requests. Properties can only be modified before sending the first request.

Generally I know that the properties need to be set before making any requests. So I considered resetting the HttpClient for each API and then just re-use it for the thousands of requests to that API. Surprisingly, I get the same error - and I have absolutely no idea why.

This is about what the code looks like:

private HttpClient ApiClient;
private List<Api> Endpoints;
[...]
foreach(Api api in this.Endpoints)
{
    this.ApiClient = new HttpClient();
    this.ApiClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(api.mediaType));
    this.ApiClient.Timeout = TimeSpan.FromMinutes(api.timeout);
    this.ApiClient.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", api.credentials);
    
    foreach (string url in api.urls)
    {
        # retrieve data from APIs and do something with it
    }
}

As mentioned earlier, the first loop works perfectly fine. But when it starts over with the second api, I get a System.InvalidOperationException with the error message above when I try to set the ApiClient's timeout value.

Why so? I have created a brand new instance of HttpClient. Is there a better way to just reset the HttpClient?

See: https://stackoverflow.com/questions/29976944/httpclient-instancing-per-service-endpoint — John H, Sep 16 '20 at 16:20
It's okay to have a few clients. That's nothing like creating a new client for each request but you can have multiple pre-configured clients and that isn't a problem. However, if you're going to use one client, set those request specific headers at the request level instead of in default headers — Aluan Haddad, Sep 16 '20 at 20:52

score 1 · Answer 1 · answered Sep 16 '20 at 16:15

1

The preferred way for generating HttpClients seems to be httpfactory: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-3.1. Also note that reinstatiating httpclients as you are, even without your specific exception can lead to problems, as your code seems to be able to run into socket exhaustion as described in https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests.

answered Sep 16 '20 at 16:15

Tarranoth

124
1
5

I will look into httpfactory. Thanks for pointing out the socked exhaustion topic, but I am aware of this - and as I am using only a very few APIs (20-40) I assume that I am safe when reusing an HttpClient for all API calls to a single API. But it would likely cause issues in a slightly different setup. – Aileron79 Sep 17 '20 at 07:29

score 0 · Answer 2 · answered Sep 16 '20 at 16:18

You should have only one instance of HttpClient during the lifetime of your application. So instead of creating a HttpClient and setting the DefaultRequestHeaders every time you loop over your endpoints use HttpRequestMessage and do the following:

this.ApiClient = new HttpClient();
foreach(Api api in this.Endpoints)
{
    HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, "url");
    
    request.Headers.Accept.Clear();
    request.Headers.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("header value"));
    var data = await ApiClient.SendAsync(request , HttpCompletionOption.ResponseContentRead);

}

I get your point, but it is thousands of requests to the same API, so this basically re-executes that part thousands of times. Most sources will tell you that it is fine to use different HttpClients for different base URLs. Still, if this turns out to be the only way that works... I'll give it a shot. — Aileron79, Sep 17 '20 at 07:25

.net HttpClient in C#: Exception when re-instantiating

2 Answers2