azure function app parallel execution ,fix httpclient instance duplicate header key issue :An item with the same key has already been added

Question

I'm using azure function apps(Queue trigger) to run the hourly jobs. I'm generating a job for each tenant and pushing it to the Queue => which triggers the function app. For each job(class with some logic) I'm passing APIservice.

APIservice is an instance of HttpClient with baseuri and tenant code will be stored in header.As can be seen below

Lets say if I have 5 tenants , 5 jobs are triggered at once and are executed in parallel. But some times (not always) I'm facing this error.

An item with the same key has already been added. Key: System.Net.Http.Headers.HeaderDescriptor

I expected each instance to be independently initialized , but sometimes it is giving this error. How do we avoid it ?

score 0 · Answer 1 · answered Apr 26 '22 at 22:11

client.DefaultRequestHeaders.Clear(); should take care of it. But it sounds like some connections are still open. Is the connection being disposed of? You could try a using statement that will help with this, or even better is to inject httpClient which is the correct way to do it. See below:

public class MyController : Controller
    {
        private readonly IConfiguration _configuration;
        private readonly HttpClient _httpClient;
        private readonly string ApiBaseURL;

        public MyController(IConfiguration configuration, HttpClient httpClient)
        {
            _configuration = configuration;
            _httpClient = httpClient;

            _httpClient.DefaultRequestHeaders.Accept.Clear();
            _httpClient.DefaultRequestHeaders.Accept.Add(
                    new MediaTypeWithQualityHeaderValue("application/json"));
            _httpClient.DefaultRequestHeaders.Add("H1", _configuration.GetValue<string>("H1"));
            _httpClient.DefaultRequestHeaders.Add("H2", _configuration.GetValue<string>("H2"));
            ApiBaseURL = _configuration.GetValue<string>("ApiBaseURL");
        }

score 0 · Answer 2 · answered Apr 28 '22 at 07:53

An item with the same key has already been added. Key: System.Net.Http.Headers.HeaderDescriptor

The Error message shows that you are adding the same headers every iteration.

To avoid that you can use the new

httpClient.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));

If you want to add the Authorization headers, you can use like below

httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", "Your Oauth token");

Normally, if you are not interested in setting up the header on the HttpClient instance by adding it to the DefaultRequestHeaders, you can use set headers per request where you'll be obliged to use the SendAsync() method.

To reuse the HttpClient, this is the right solution and good practice to:

Avoid Performance and port exhaustion problems
Avoid adding the same Headers on each iteration
Make headers thread-safe.

azure function app parallel execution ,fix httpclient instance duplicate header key issue :An item with the same key has already been added

2 Answers2

References: