C# HttpClient custom headers each request

Question

I've noticed that using HttpClient is NOT thread safe when modifying HttpClient.DefaultRequestHeaders but I want to make as many requests as possible. I need a custom header each request (2 other headers are always the same). Also the URL changes a bit

1. http://example.com/books/1234/readers/837
2. http://example.com/books/854/readers/89
3. http://example.com/books/29432/readers/238
4. ... so on

Currently I'm creating a new HttpClient for every request but I feel like creating 10k+ HttpClients isn't the best choice here.

I'd like to make one static HttpClient with 2 DefaultRequestHeaders and use this HttpClient for every request but also add one custom header.

I want to make this as fast as possible so if you have something else I'll take it.

        Parallel.ForEach(Requests, Request =>
        {
            var Client = new HttpClient();
            Client.DefaultRequestHeaders.Clear();
            Client.DefaultRequestHeaders.Add("Header1", "Value1");
            Client.DefaultRequestHeaders.Add("Header2", "Value2");
            Client.DefaultRequestHeaders.Add("Header3", "Value for exact this request");

            var response = Client.PutAsync(new Uri($"http://example.com/books/1234/readers/837"), null); //.Result (?)
            Client.Dispose();
        });

Answer 1

Don’t use DefaultRequestHeaders for headers that don’t apply to all requests the HttpClient sends.

Also, don’t create an HttpClient per request.

You can do this easily by instead creating one HttpRequestMessage for each request, applying whatever headers you need to it, and using the same HttpClient throughout to .SendAsync() them:

using (var request = new HttpRequestMessage(HttpMethod.Put, url)
{
    request.Headers.<add here>;
    // optionally set .Content

    using (var response = await httpClient.SendAsync(request))
    {
        // ... process response
    }
}

Answer 2

The standard scenario of using HttpClient since .NET Core 2.1 is:

• create it via HttpClientFactory
• custom headers passed in the body of queries (fixed headers can be defined in custom DelegatingHandler)

Straightforward example with direct passing all headers and without batch-processing:

var tasks = Enumerable.Range(1, 10000)
    .Select(v => HttpClientFactory.Create().SendAsync(new HttpRequestMessage(HttpMethod.Put, new Uri($"http://example.com/books/1234/readers/837"))
    {
        Headers =
        {
            {"Header1", "Value1"},
            {"Header2", "Value2"},
            {"Header3", v.ToString()},
        },
        Content = new StringContent("{test:\"hello\"}", Encoding.UTF8, "application/json")
    }));

var responses = await Task.WhenAll(tasks).ConfigureAwait(false);
// ..

Remarks:

• sending thousands request looks suspicious, are you sure that server doesn't have the endpoint for group processing of items
• using PLINQ for the non-CPU consuming tasks isn't correct (quotation: "PLINQ queries scale in the degree of concurrency based on the capabilities of the host computer")

Answer 3

I think the usage of a DelegatingHandler would be able to accomplish what you are looking for. You would stop using DefaultRequestHeaders and instead set them within the DelegatingHandler.

Also note, HttpClient is threadsafe for requests, and should not be disposed of. This is because it disposes of the underlying HttpMessageHandler which causes some lower level inefficiencies. More detail is here http://www.nimaara.com/2016/11/01/beware-of-the-net-httpclient/.

Sample.cs

// .Net Framework, careful here because the sockets will be cached, network changes will break your app
// see http://www.nimaara.com/2016/11/01/beware-of-the-net-httpclient/ for more details

var client = new HttpClient(new MyCustomHandler() { InnerHandler = new HttpClientHandler() });

// .Net Core
client = new HttpClient(new MyCustomHandler() { InnerHandler = new SocketsHttpHandler() { PooledConnectionLifetime = TimeSpan.FromMinutes(1) } });

Parallel.ForEach(Requests, Request =>
{

    var response = client.PutAsync(new Uri($"http://example.com/books/1234/readers/837"), null); //.Result (?)

    // do not dispose of httpclient!
});

MyCustomHandler.cs

public class MyCustomHandler : DelegatingHandler
{
    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        // the static headers
        request.Headers.Add("Header1", "Value1");
        request.Headers.Add("Header2", "Value2");

        // the unique header
        SignRequest(request);

        return base.SendAsync(request, cancellationToken);
    }


    public void SignRequest(HttpRequestMessage message)
    {
        // usually the pattern of a unique header per request is for an authorization header based on some signature of the request
        // this logic would be here
        // generate the signature
        string signature = new Random().Next(int.MaxValue).ToString();

        message.Headers.Add("Header3", signature);
    }
}