How can I emulate a web browser http request from code?

Question

I use C# in my WPF Project. I want to send a GET http request to a website, but I want to send it in a way, so that it will look like a request from a browser.
Now I have a program which sends a GET request and gets a response. I use WebRequest class for sending GET requests.
I know that browsers add some information to their requests like browser name, OS name and the computer name.
My question is how can I add this information to my WebRequest? To what properties all that information (browser name, OS name) should be assigned?

Answer 1

You should use Fiddler to capture the request that you want to simulate. You need to look at the inspectors > raw. This is an example of a request to the fiddler site from chrome

GET http://fiddler2.com/ HTTP/1.1
Host: fiddler2.com
Connection: keep-alive
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.94 Safari/537.36
Referer: https://www.google.be/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,nl;q=0.6

You can then set each one of these headers in your webrequest (see http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx).

WebRequest request = (HttpWebRequest)WebRequest.Create("http://www.test.com");      
request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.94 Safari/537.36";

Answer 2

Generally the information you are interested in (browser, os, etc.) is sent in the "User Agent" header along with the request. You can control the user agent with its property, here:

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.useragent.aspx

There may be other differences, I recommend using Fiddler to capture your browser traffic and then compare it to the traffic from your .NET-based web request.

http://fiddler2.com/

Enjoy.

Answer 3

All such information is sent via header in a web request. You can also add such information in header as key/value pair. However, you have only limited attributes which you can set using WebRequest's header property; many of them are restricted. You can also check the list of restricted header attributes in the following thread: Cannot set some HTTP headers when using System.Net.WebRequest.