cURL request with HttpClient - how to set the timeout on the server connection (WinInet)

Question

I am sending cURL request using HttpClient through the method described here under.

The parameter used for this method are:

SelectedProxy = a custom class that stores my proxy's parameters

Parameters.WcTimeout = the timeout

url, header, content = the cURL request (based on this tool to convert to C# https://curl.olsh.me/).

        const SslProtocols _Tls12 = (SslProtocols)0x00000C00;
        const SecurityProtocolType Tls12 = (SecurityProtocolType)_Tls12;
        ServicePointManager.SecurityProtocol = Tls12;
        string source = "";

        using (var handler = new HttpClientHandler())
        {
            handler.UseCookies = usecookies;
            WebProxy wp = new WebProxy(SelectedProxy.Address);
            handler.Proxy = wp;

            using (var httpClient = new HttpClient(handler))
            {
                httpClient.Timeout = Parameters.WcTimeout;

                using (var request = new HttpRequestMessage(new HttpMethod(HttpMethod), url))
                {
                    if (headers != null)
                    {
                        foreach (var h in headers)
                        {
                            request.Headers.TryAddWithoutValidation(h.Item1, h.Item2);
                        }
                    }
                    if (content != "")
                    {
                        request.Content = new StringContent(content, Encoding.UTF8, "application/x-www-form-urlencoded");
                    }

                    HttpResponseMessage response = new HttpResponseMessage();
                    try
                    {
                        response = await httpClient.SendAsync(request);
                    }
                    catch (Exception e)
                    {
                        //Here the exception happens
                    }
                    source = await response.Content.ReadAsStringAsync();
                }
            }
        }
        return source;

If I am running this without proxy, it works like a charm. When I send a request using a proxy which I tested first from Chrome, I have the following error on my try {} catch {}. Here is the error tree

{"An error occurred while sending the request."}
    InnerException {"Unable to connect to the remote server"}
        InnerException {"A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond [ProxyAdress]"}
        SocketErrorCode: TimedOut

By using a Stopwatch I see that the TimedOut occurred after around 30 sec.

---

I tried a few different handler based on the following links What's the difference between HttpClient.Timeout and using the WebRequestHandler timeout properties?, HttpClient Timeout confusion or with the WinHttpHandler.

It's worth noting that WinHttpHandler allow for a different error code, i.e. Error 12002 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'The operation timed out'. The underlying reason is the same though it helped to target where it bugs (i.e. WinInet) which confirms also what @DavidWright was saying regarding that timeouts from HttpClient manages a different part of the request sending.

Hence my issue is coming from the time it takes to establish a connection to the server, which triggers the 30sec timeout from WinInet.

My question is then How to change those timeout?

On a side note, it's worth noting that Chrome, which uses WinInet, does not seem to suffer from this timeout, nor Cefsharp on which a big part of my app is based, and through which the same proxies can properly send requests.

Answer 1

So thanks to @DavidWright I understand a few things:

1. Before that the HttpRequestMessage is sent and the timeout from HttpClient starts, a TCP connection to the server is initiated
2. The TCP connection has its own timeout, defined at OS level, and we do not identified a way to change it at run time from C# (question pending if anyone want to contribute)
3. Insisting on trying to connect works as each try benefits from previous tries, though proper exception management & manual timeout counter needs to be implemented (I actually considered a number of tries in my code, assuming each try is around 30sec)

All this together ended up in the following code:

        const SslProtocols _Tls12 = (SslProtocols)0x00000C00;
        const SecurityProtocolType Tls12 = (SecurityProtocolType)_Tls12;
        ServicePointManager.SecurityProtocol = Tls12;
        var sp = ServicePointManager.FindServicePoint(endpoint);

        sp.ConnectionLeaseTimeout = (int)Parameters.ConnectionLeaseTimeout.TotalMilliseconds;


        string source = "";

        using (var handler = new HttpClientHandler())
        {
            handler.UseCookies = usecookies;
            WebProxy wp = new WebProxy(SelectedProxy.Address);
            handler.Proxy = wp;

            using (var client = new HttpClient(handler))
            {
                client.Timeout = Parameters.WcTimeout;

                int n = 0;
                back:
                using (var request = new HttpRequestMessage(new HttpMethod(HttpMethod), endpoint))
                {

                    if (headers != null)
                    {
                        foreach (var h in headers)
                        {
                            request.Headers.TryAddWithoutValidation(h.Item1, h.Item2);
                        }
                    }
                    if (content != "")
                    {
                        request.Content = new StringContent(content, Encoding.UTF8, "application/x-www-form-urlencoded");
                    }
                    HttpResponseMessage response = new HttpResponseMessage();

                    try
                    {
                        response = await client.SendAsync(request);
                    }
                    catch (Exception e)
                    {
                        if(e.InnerException != null)
                        {
                            if(e.InnerException.InnerException != null)
                            {
                                if (e.InnerException.InnerException.Message.Contains("A connection attempt failed because the connected party did not properly respond after"))
                                {
                                    if (n <= Parameters.TCPMaxTries)
                                    {
                                        n++;
                                        goto back;
                                    }
                                }
                            }
                        }
                        // Manage here other exceptions
                    }
                    source = await response.Content.ReadAsStringAsync();
                }
            }
        }
        return source;

On a side note, my current implementation of HttpClient may be problematic in the future. Though being disposable, HttpClient should be defined at App level through a static, and not within a using statement. To read more about this go here or there.

My issue is that I want to renew the proxy at each request and that it is not set on a per request basis. While it explains the reasdon of the new ConnectionLeaseTimeout parameter (to minimize the time the lease remains open) it is a different topic

Answer 2

I have had the same problem with HttpClient. Two things need to happen for SendAsync to return: first, setting up the TCP channel over which the communication occurs (the SYN, SYN/ACK, ACK handshake, if you're familiar with that) and second getting back the data that constitutes the HTTP response over that TCP channel. HttpClient's timeout only applies to the second part. The timeout for the first part is governed by the OS's network subsystem, and it's quite difficult to change that timeout in .NET code.

(Here's how you can reproduce this effect. Set up a working client/server connection between two machines, so you know that name resolution, port access, listening, and client and server logic all works. Then unplug the network cable on the server and re-run the client request. It will time out with the OS's default network timeout, regardless of what timeout you set on your HttpClient.)

The only way I know around this is to start your own delay timer on a different thread and cancel the SendAsync task if the timer finishes first. You can do this using Task.Delay and Task.WaitAny or by creating a CancellationTokenSource with your desired timeone (which essentially just does the first way under the hood). In either case you will need to be careful about cancelling and reading exceptions from the task that loses the race.