System.Net.WebRequest not respecting hosts file

Question

Is there a way to get a System.Net.WebRequest or System.Net.WebClient to respect the hosts or lmhosts file?

For example: in my hosts file I have:

10.0.0.1  www.bing.com

When I try to load Bing in a browser (both IE and FF) it fails to load as expected.

Dns.GetHostAddresses("www.bing.com")[0]; // 10.0.0.1 
WebRequest.Create("http://10.0.0.1").GetResponse(); // throws exception (expected)
WebRequest.Create("http://www.bing.com/").GetResponse(); // unexpectedly succeeds

Similarly:

WebClient wc = new WebClient();
wc.DownloadString("http://www.bing.com"); //succeeds 

Why would System.Net.Dns respect the hosts file but System.Net.WebRequest ignore it? What do I need to change to make the WebRequest respect the hosts file?

Additional Info:

• If I disable IPv6 and set my IPv4 DNS Server to 127.0.0.1, the above code works (fails) as expected. However if I add my normal DNS servers back as alternates, the unexpected behavior resumes.
• I've reproduced this on 3 Win7 and 2 Vista boxes. The only constant is my company's network.
• I'm using .NET 3.5 SP1 and VS2008

Edit

Per @Richard Beier's suggestion, I tried out System.Net tracing. With tracing ON the WebRequest fails as it should. However as soon as I turn tracing OFF the behavior reverts to the unexpected success. I have reproduced this on the same machines as before in both debug and release mode.

Edit 2

This turned out to be the company proxy giving us issues. Our solution was a custom proxy config script for our test machines that had "bing.com" point to DIRECT instead of the default proxy.

Answer 1

I think that @Hans Passant has spotted the issue here. It looks like you have a proxy setup in IE.

Dns.GetHostAddresses("www.bing.com")[0]; // 10.0.0.1  

This works because you are asking the OS to get the IP addresses for www.bing.com

WebRequest.Create("http://www.bing.com/").GetResponse(); // unexpectedly succeeds

This works because you are asking the framework to fetch a path from a server name. The framework uses the same engine and settings that IE frontend uses and hence if your company has specified by a GPO that you use a company proxy server, it is that proxy server that resolves the IP address for www.bing.com rather than you.

WebRequest.Create("http://10.0.0.1").GetResponse(); // throws exception (expected) 

This works/fails because you have asked the framework to fetch you a webpage from a specific server (by IP). Even if you do have a proxy set, this proxy will still not be able to connect to this IP address.

I hope that this helps.

Jonathan

Answer 2

I'm using VS 2010 on Windows 7, and I can't reproduce this. I made the same hosts-file change and ran the following code:

Console.WriteLine(Dns.GetHostAddresses("www.bing.com")[0]);             // 10.0.0.1     
var response = WebRequest.Create("http://www.bing.com/").GetResponse(); // * * *
Console.WriteLine(new StreamReader(response.GetResponseStream()).ReadToEnd());

I got an exception on the line marked "* * *". Here's the exception detail:

System.Net.WebException was unhandled
  Message=Unable to connect to the remote server
  Source=System
  StackTrace:
       at System.Net.HttpWebRequest.GetResponse()
       at ConsoleApplication2.Program.Main(String[] args) in c:\Data\Projects\ConsoleApplication2\ConsoleApplication2\Program.cs:line 17
  InnerException: System.Net.Sockets.SocketException
       Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.0.0.1:80
       Source=System
       ErrorCode=10060

Maybe it's an issue with an earlier .NET version, that's now fixed in .NET 4 / VS 2010? Which version of .NET are you using?

I also found this thread from 2007, where someone else ran into the same problem. There are some good suggestions there, including the following:

• Turn on system.net tracing

• Work around the problem by using Dns.GetHostAddresses() to resolve it to an IP. Then put the IP in the URL - e.g. "http://10.0.0.1/". That may not be an option for you though.

In the above thread, mariyaatanasova_msft also says: "HttpWebRequest uses Dns.GetHostEntry to resolve the host, so you may get a different result from Dns.GetHostAddresses".

Answer 3

You should overwrite the default proxy. HttpWebRequest & WebRequest will set a default proxy if present in Internet Explorer and your file hosts will be bypassed.

request.Proxy = new WebProxy();

The following is just an example of code:

try
{
   HttpWebRequest request = (HttpWebRequest)WebRequest.Create("www.bing.com");
   request.Proxy = new WebProxy();
   request.Method = "POST";            
   request.AllowAutoRedirect = false;

   HttpWebResponse response = (HttpWebResponse)request.GetResponse();               
   if (response.StatusCode == HttpStatusCode.OK)
   {
      //some code here
   }
}
catch (exception e)
{
   //Some other code here
}