1

I have the following code for checking availability applications from play market and app store:

public async Task<List<CheckPackageResponse>> CheckAsync(List<string> appIds)
{
    var sem = new SemaphoreSlim(200);

    var tasks = appIds.Select(async (a, i) =>
    {
        await sem.WaitAsync();

        var response = new CheckPackageResponse
        {
            AppId = a,
        };

        string url;

        // if IOS
        if (a.All(Char.IsDigit))
        {
            url = $"https://apps.apple.com/ru/app/id{a}?dataOnly=true&isWebExpV2=true";
        }
        else
        {
            url = $"https://play.google.com/store/apps/details?id={a}";
        }

        try
        {
            _client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36");

            var resp = await _client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
            response.HttpStatus = (int) resp.StatusCode;

            return response;
        }
        catch (Exception e)
        {
            response.ErrorMessage = e.Message;
            response.StackTrace = e.StackTrace;

            return response;
        }
        finally
        {
            sem.Release();
        }

    }).ToList();

    return (await Task.WhenAll(tasks)).ToList();
}

And in the Startup.cs I have the following to inject HttpClient:

services.AddHttpClient<ICheckPackagesService, CheckPackagesService>(x =>
{
    x.Timeout = TimeSpan.FromSeconds(7);
}).ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
{
    Proxy = new WebProxy(Configuration["Proxy"] != "" ? Configuration["Proxy"] : null)
});

It works nice. I can give list of application ids and API will answer me is an application available or not. But it has some problem. After some time it has crash with error

Address in use.

It tells me that HTTP client does not close connections. But I know that this problem possible if I use something like this:

using(var http = new HttpClient(...)) {
    ... 
}

And it's definitely not my case. So what is wrong in my case? Where is the mistake?

Update

I will pin here my stack trace

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)\n at System.Threading.Tasks.ValueTask1.get_Result()\n at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n at System.Threading.Tasks.ValueTask1.get_Result()\n at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask1 creationTask)\n at System.Threading.Tasks.ValueTask1.get_Result()\n at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)\n at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\n at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\n at ApChecker.Services.Concrete.CheckPackagesService.<>c__DisplayClass2_0.<b__0>d.MoveNext() in /builds/mobilemediaru/makeapp/apchecker/ApChecker/Services/Concrete/CheckPackagesService.cs:line 50

Aleksej_Shherbak
  • 2,757
  • 5
  • 34
  • 71
  • 1
    Please check this https://stackoverflow.com/questions/15705092/do-httpclient-and-httpclienthandler-have-to-be-disposed – Jayakumar Thangavel Nov 03 '19 at 10:30
  • *After some time* is too broad, can you clarify or provide more information then that? what is the actual exception thrown here? is it a Socket exception? what raises it? what does the stack trace show? – Brett Caswell Nov 03 '19 at 10:37
  • @BrettCaswell thank you for comment. I have updated me question. But is it useful? I use my http client in the async cosntruction like `appIds.Select(async (a, i) => { ... }`. And it gives me common exception in this case, that tells about canceling operation. – Aleksej_Shherbak Nov 03 '19 at 10:44
  • hmm.. this issue could relate to Socket Port exhaustion... where the cause could relate to initializing new instances of HttpClient (irregardless to the disposal of them). – Brett Caswell Nov 03 '19 at 10:55
  • But I use only one http client instance, am I right? Via http client factory. Like in this tutorial https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests#implement-your-typed-client-classes-that-use-the-injected-and-configured-httpclient – Aleksej_Shherbak Nov 03 '19 at 10:59
  • 1
    *it will receive a new HttpClient instance each time it's constructed* in that link you provided – Brett Caswell Nov 03 '19 at 11:03
  • 1
    Listen guys, in the link that @JayakumarThangavel have pined in the comment above I have found information about sending `Connection : close` HTTP header. Maybe it will solve my problem. What do you think? – Aleksej_Shherbak Nov 03 '19 at 11:04
  • @BrettCaswell hmm, yes, you are right. I thought that It will create only one HTTP client instance for whole application. But now I see that it is false. – Aleksej_Shherbak Nov 03 '19 at 11:08
  • I feel like there is more to this then starving the connection pool though.. the concurrency in this operation along with the retry is a consideration here. I'm not sure how relevant this [github bug](https://github.com/dotnet/corefx/issues/37044) issue is but it has me curious if some internals are working correctly here – Brett Caswell Nov 03 '19 at 11:11
  • 1
    I read more of that documentation, you're right.. in fact `HttpMessageHandler` objects are managing the pool of each `HttpClient`, so it doesn't really matter about the initialization of new `HttpClient` as much as the configurable notion on `HttpMessageHandler` lifetimes. – Brett Caswell Nov 03 '19 at 11:36
  • Last question from me, how many `appIds` records are you running? – Brett Caswell Nov 03 '19 at 11:37
  • @BrettCaswell just one `app_id`. – Aleksej_Shherbak Nov 03 '19 at 11:54
  • 1
    I have added ```Connection: close``` HTTP header, and I have added ```Dispose()``` call for the HTTP client response. I hope it will solve my issue. If it works, I will publish it as answer. – Aleksej_Shherbak Nov 03 '19 at 13:50

0 Answers0