HttpWebRequests using WebProxy work and then fail after time

Question

Odd situation I have here and unfortunately I don't understand a lot about the Windows network side of things outside of netstat:

So I have a proxy that I have configured in my browser (Firefox 42) and am running a simple application that loops through URLs to call them via that proxy. This proxy has credentials in order to use it and I know the proxy works. This is a Windows 7 box.

So at some point during this process, the following happens:

Browser calls just time out. It doesn't ask for credentials at all. (when the issue goes away, it starts to ask for credentials again).
Calls in the application timeout no matter what the timeout is (7 seconds, 20 seconds, etc)

I've confirmed the following:

In my .net application, I 100% know I am closing every network object and am even aborted the request object after I read the response.
After a certain amount of time, without any calls, the problem goes away.
When I use this proxy on another server, it 100% works. So I know it's related to the server I am using and that proxy IP address.
I've looked at the resource manager and there aren't a lot of active TCP connections open. Although I don't know if that means anything.
If I use another proxy, THAT proxy works. It's like it's IP specific, which is baffling me because it's just a web proxy object in the code.

What would cause this? It usually happens after 4-7 calls with the proxy and releases the issue after 30-40 minutes.

Edit 7:

Also happens with AWS instances. Tried that approach. zzz...

Edit 6:

Doesn't go away with a server restart either. You can restart and 15 minutes later SAME proxy times out. Eventually works again.

Edit 5:

Wrote a similar test with Java and Python. Same result.

Edit 4: This is how it works:

Call to Proxy 1... Good!
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Good!
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Timeout...
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!
Call to Proxy 1... Good!
Call to Proxy 2... Good!
Call to Proxy 3... Good!
Call to Proxy 4... Good!

Edit 3: These questions appears to be very similar: Http Post WebRequest getting timed out

HttpWebRequest and GetResponse hangs after 1 request; other solutions doesn't work for me

WebRequest.GetResponse locks up?

HttpWebRequest times out on second call

Edit 2: Looking at Wireshark, I'm seeing a TCP transmission in the info for the proxy affected. But doesn't that happen with other proxies at the same time? So is that coming from the proxy server itself? It doesn't make sense to me since I am not even getting a response back and the request isn't even being processed.

Edit: Adding code for calls in code. This method is called in a while loop over and over again:

            String html = null;

        HttpWebRequest request = null;
        WebProxy webProxy = null;

        try
        {
            request = (HttpWebRequest)WebRequest.Create(url);

            webProxy = new WebProxy(proxyIP, proxyPort);
            webProxy.Credentials = new NetworkCredential(proxyUser, proxyPass);

            request.Proxy = webProxy;
            request.KeepAlive = false;
            request.Timeout = 5000;
            request.ReadWriteTimeout = 5000;
            request.Method = "GET";
            request.UserAgent = generateAgentString();

            using (WebResponse resp = (WebResponse)request.GetResponse())
            {
                using (Stream strm = resp.GetResponseStream())
                {
                    StreamReader reader = new StreamReader(strm, Encoding.UTF8);

                    try
                    {
                        html = reader.ReadToEnd();
                    }
                    catch
                    {
                        Console.WriteLine("Failed");
                        html = null;
                    }
                    finally
                    {
                        strm.Flush();
                        reader.BaseStream.Dispose();
                        reader.Dispose();
                        strm.Dispose();
                        resp.Dispose();
                    }
                }
            }

            if (request != null)
            {
                request.Abort();
            }
        }
        catch(Exception e) { Console.WriteLine(e); }

A number of things could be happening. You could have a collisions on your network or a faulty switch so the packet never reach the proxy or rather any problem on the way to the host you're talking to. I would like to know what the actual exception you get is. And inner exceptions if you have any. Secondly if you can capture a failed request in wireshark that would tell what's happening. — fhogberg, Jan 27 '16 at 23:39
So ignoring your code, when you setup your actual browser to use this proxy it still times out occasionally? Are you writing the code for this proxy, and you're trying to determine why it's having issues? It sounds like a problem with the code in the proxy server not managing network resources properly. — Brett Allen, Jan 28 '16 at 05:31
Yeah - I've set up the proxy in the browser BOTH on the server the code is running (which results in a timeout as well) and on another server outside of the network (which results in a successful call). — user2124871, Jan 28 '16 at 13:21
I would suspect that this is related to the network or the server itself rather then the proxy as it works from the other server. If the proxy server was having a problem both servers should see it. Sometimes the easiest thing is to try to move the server, change network cable, etc and try to elliminate a thing at the time. Go big first. Move it next to the working server with a brand new network cable. If that works its just a matter of moving it back changing a thing at a time until the problem appears again. If it doesn't you know for sure it's the server that is the problem. — fhogberg, Jan 28 '16 at 19:20
Add this code int limit = ServicePointManager.DefaultConnectionLimit; — Viru, Feb 01 '16 at 19:03
This will tell you if there any kind of limit on number of http request you can create...and if there is you can override it with value you want — Viru, Feb 01 '16 at 19:04
It's just odd to me that it would be proxy server specific though, wouldn't that resist connections at a higher level? — user2124871, Feb 02 '16 at 21:52

score 1 · Accepted Answer · answered Feb 18 '16 at 15:47

After much digging - I went back to the proxy provider again with a bunch of trace information and digging results and... miraculously... they've decided that they do in fact have limitations that they have not disclosed to the purchaser.

Shocker. I'm more annoyed I went this far after initially discussing with them and getting a clean bill a health on their end.

If anyone needs a recommendation for a provider not to use :)

HttpWebRequests using WebProxy work and then fail after time

1 Answers1