HttpWebRequest doesn't work except when fiddler is running

Question

This is probably the weirdest problem I have run into. I have a piece of code to submit POST to a url. The code doesn't work neither throws any exceptions when fiddler isn't running, However, when fiddler is running, the code posts the data successfuly. I have access to the post page so I know if the data has been POSTED or not. This is probably very non-sense, But it's a situation I am running into and I am very confused.

byte[] postBytes = new ASCIIEncoding().GetBytes(postData);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://myURL);
req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10";
req.Accept = "application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
req.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3");
req.Headers.Add("Accept-Language", "en-US,en;q=0.8");
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = postBytes.Length;
req.CookieContainer = cc;
Stream s = req.GetRequestStream();
s.Write(postBytes, 0, postBytes.Length);
s.Close();

Answer 1

If you don't call GetResponseStream() then you can't close the response. If you don't close the response, then you end up with a socket in a bad state in .NET. You MUST close the response to prevent interference with your later request.

Answer 2

Close the HttpWebResponse after getting it.

I had the same problem, then I started closing the responses after each request, and Boom, no need to have fiddler running.

This is a pseudo of a synchronous code:

request.create(url);

///codes

httpwebresponse response = (httpwebresponse)request.getresponse();

/// codes again like reading it to a stream

response.close();

Answer 3

I had a similar problem recently. Wireshark would show the HTTPWebRequest not leave the client machine unless Fiddler was running. I tried removing proxy settings, but that didn't fix the problem for me. I tried everything from setting the request to HttpVersion.Version10, enabling/disabling SendChuck, KeepAlive, and a host of other settings. None of which worked.

Ultimately, I just checked if .Net detected a proxy and had the request attempt to ignore it. That fixed my issue with request.GetResponse() throwing an immediate exception.

IWebProxy proxy = request.Proxy;

if (request.Proxy != null)
{
    Console.WriteLine("Removing proxy: {0}", proxy.GetProxy(request.RequestUri));
    request.Proxy = null;
}

Answer 4

In my case when I had the same situation (POST only works when Fiddler is running) the code was sending the POST from an application running on IISExpress in a development environment behind a proxy to an external server. Apparently even if you have proxy settings configured in Internet Options the environment IIS is running in may not have access to them. In my work environment I simply had to update web.config with the path to our proxy's configuration script. You may need to tweak other proxy settings. In that case your friend is this MSDN page that explains what they are: http://msdn.microsoft.com/en-us/library/sa91de1e.aspx.

Ultimately I included the following in the application's web.config and then the POST went through.

<configuration>
  <system.net>
    <defaultProxy>
      <proxy scriptLocation="http://example.com:81/proxy.script" />
    </defaultProxy>
  </system.net>
</configuration>

Answer 5

I ran into the same problem with Python - requests to a local server were failing with a 404, but then when I ran them with Fiddler running they were working correctly.

The real clue to the problem here is that Fiddler works by acting as a proxy for HTTP traffic so that all requests from the local machine go through Fiddler rather than straight out into the network.

In the exact situation I was in, I was making requests to a local server, regular traffic passes through a proxy and in Local Area Network (LAN) Settings for the network connection the Proxy server pane the Bypass proxy server for local addresses option was checked.

My suspicion is that the "Bypass proxy server for local addresses" is not necessarily picked up by the programming language, but the proxy server details are. Fiddler is aware of that policy, so requests through Fiddler work but requests direct from the programming language don't.

By setting the proxy for the request for the local server to nothing, it worked correctly from code. Obviously, that could be a gotcha if you find yourself moving from an internal to external server during deployment.

Answer 6

I faced the same scenario : I was POSTing to an endpoint behind Windows Authentication.

Fiddler keeps a pool of open connections, but your C# test or powershell script does not when it runs without fiddler.

So you can make the test/script to also maintain a pool of open authenticated connections, by setting the property UnsafeAuthenticatedConnectionSharing to true on your HttpWebRequest. Read more about it here, microsoft KB. In both cases in that article, you can see that they are making two requests. The first one is a simple GET or HEAD to get the authentication header (to complete the handshake), and the second one is the POST, that will use the header obtained before.

Apparently you cannot (sadness) directly do the handshake with POST http requests.

Answer 7

Always use using construct. it make sure all resource release after call

   using (HttpWebResponse responseClaimLines = (HttpWebResponse)requestClaimLines.GetResponse())
                {
                    using (StreamReader reader = new StreamReader(responseClaimLines.GetResponseStream()))
                    {
                        responseEnvelop = reader.ReadToEnd();
                    }
                }

add following entries to webconfig file

<system.net>
<connectionManagement>
<add address="*" maxconnection="30"/>

Answer 8

Well i have faced similar problem few weeks back and the reason was that when fiddler is running it changes the proxy settings to pass the request through Fiddler but when its closed the proxy somehow still remains and thus doesn't allow your request to go ahead on internet.

I tried by setting the IE's and Firefox's network settings to not to take any proxy and it worked.

Try this, may it be the same problem...

Answer 9

I found the solution in increasing the default number of connections

ServicePointManager.DefaultConnectionLimit = 10000;