How to interact with a website without a browser?

Question

Say I am building a c# application. The purpose of application to :

get username & password from user.
and show some information present on the website.

in the background, after taking username and password, it should :

log in to a website with those credentials.
and click on the anchor link that appears after logging in.
find out the span that hold the info.
get the info.

that was an example. I am actually building an app to show bandwidth usage information. The server does not expose any API for that.

Is there any tutorial/info/article available for similar purpose ? I just don't what to search for ?

This isn't what your actual question asks, but to "interact with a website without a browser" you can use lynx - technically a browser, but it is a terminal application. Not sure if it is available on Windows though ... — NickO, Dec 22 '12 at 05:01
There are classes like `HTTPwebRequest` and `HTTPwebResponse` which can be use but in console application I don't know. And in web to it is difficult to simulate the click event. here is a link http://forums.asp.net/t/1439779.aspx/1 — शेखर, Dec 22 '12 at 05:07
you have the answer in your tags, httpwebrequest/response, or the webClient wrapper — Steve, Dec 22 '12 at 05:07
@krshekhar, it doesn't have to be a console application. just needs to be c# — Bilal Fazlani, Dec 22 '12 at 05:09

score 18 · Accepted Answer · answered Dec 22 '12 at 06:12

Basic Introduction To HttpWebRequests

Firstly, you're going to need the right tools for the job. Go and download the Live HTTP Headers plugin for Firefox. This will allow you to view HTTP headers in real time so you can view the POST data that is sent when you interact with the website. Once you know the data that is sent to the website you can emulate the process by creating your own HTTP web requests programmatically. Tool > Live HTTP Headers

Load Live HTTP Headers by navigating to Tools > Live HTTP Headers. Once you've loaded the GUI navigate to the website you wish to login to, I will use Facebook for demonstration purposes. Type in your credentials ready to login, but before you do Clear the GUI text window and ensure that the check box labeled Capture is checked. Once you hit login you will see the text window flood with various information about the requests including the POST data which you need.

I find it best to click Save All... and then search for your username in the text document so that you can identify the POST data easily. For my request the POST data looked like this:

lsd=AVp-UAbD&display=&legacy_return=1&return_session=0&trynum=1&charset_test=%E2%82%AC%2C%C2%B4%2C%E2%82%AC%2C%C2%B4%2C%E6%B0%B4%2C%D0%94%2C%D0%84&timezone=0&lgnrnd=214119_mDgc&lgnjs=1356154880&email=%myfacebookemail40outlook.com&pass=myfacebookpassword&default_persistent=0

Which can then be defined in C# like so:

StringBuilder postData = new StringBuilder();
postData.Append("lsd=AVqRGVie&display=");
postData.Append("&legacy_return=1");
postData.Append("&return_session=0");
postData.Append("&trynum=1");
postData.Append("&charset_test=%E2%82%AC%2C%C2%B4%2C%E2%82%AC%2C%C2%B4%2C%E6%B0%B4%2C%D0%94%2C%D0%84");
postData.Append("&timezone=0");
postData.Append("&lgnrnd=153743_eO6D");
postData.Append("&lgnjs=1355614667");
postData.Append(String.Format("&email={0}", "CUSTOM_EMAIL"));
postData.Append(String.Format("&pass={0}", "CUSTOM_PASSWORD"));
postData.Append("&default_persistent=0");

I'm aiming to show you the relation between the POST data that we can send 'manually' via the web browser and how we can use said data to emulate the request in C#. Understand that sending POST data is far from deterministic. Different websites work in different ways and can throw all kinds of things your way. Below is a function I put together to validate that Facebook credentials are correct. I can't and shouldn't go into extraordinary depth here as the classes and their members are well self-documented. You can find better information than I can offer about the methods used at MSDN for example, WebRequest.Method Property

    private bool ValidateFacebookCredentials(string email, string password)
    {
        CookieContainer cookies = new CookieContainer();
        HttpWebRequest request = null;
        HttpWebResponse response = null;
        string returnData = string.Empty;

        //Need to retrieve cookies first
        request = (HttpWebRequest)WebRequest.Create(new Uri("https://www.facebook.com/login.php?login_attempt=1"));
        request.Method = "GET";
        request.CookieContainer = cookies;
        response = (HttpWebResponse)request.GetResponse();

        //Set up the request
        request = (HttpWebRequest)WebRequest.Create(new Uri("https://www.facebook.com/login.php?login_attempt=1"));
        request.Method = "POST";
        request.ContentType = "application/x-www-form-urlencoded";
        request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13";
        request.Referer = "https://www.facebook.com/login.php?login_attempt=1";
        request.AllowAutoRedirect = true;
        request.KeepAlive = true;
        request.CookieContainer = cookies;

        //Format the POST data
        StringBuilder postData = new StringBuilder();
        postData.Append("lsd=AVqRGVie&display=");
        postData.Append("&legacy_return=1");
        postData.Append("&return_session=0");
        postData.Append("&trynum=1");
        postData.Append("&charset_test=%E2%82%AC%2C%C2%B4%2C%E2%82%AC%2C%C2%B4%2C%E6%B0%B4%2C%D0%94%2C%D0%84");
        postData.Append("&timezone=0");
        postData.Append("&lgnrnd=153743_eO6D");
        postData.Append("&lgnjs=1355614667");
        postData.Append(String.Format("&email={0}", email));
        postData.Append(String.Format("&pass={0}", password));
        postData.Append("&default_persistent=0");

        //write the POST data to the stream
        using(StreamWriter writer = new StreamWriter(request.GetRequestStream()))
            writer.Write(postData.ToString());

        response = (HttpWebResponse)request.GetResponse();

        //Read the web page (HTML) that we retrieve after sending the request
        using (StreamReader reader = new StreamReader(response.GetResponseStream()))
            returnData = reader.ReadToEnd();

        return !returnData.Contains("Please re-enter your password");
    }

after login successful with this code. Can one go to the browser and navigate within facebook without entering the credentials in the browser again? I mean, does the user session is kept active as if we login directly through the browser? — Nahum Fabian, Nov 07 '14 at 02:46

score 2 · Answer 2 · answered Dec 22 '12 at 05:16

Sample Code on Grabbing Contents (Screen Scraping)

 Uri uri = new Uri("http://www.microsoft.com/default.aspx");
 if(uri.Scheme = Uri.UriSchemeHttp) 
 {
     HttpWebRequest request = HttpWebRequest.Create(uri);
     request.Method = WebRequestMethods.Http.Get;
     HttpWebResponse response = request.GetResponse();
     StreamReader reader = new StreamReader(response.GetResponseStream());
     string  tmp = reader.ReadToEnd();
     response.Close();
     Response.Write(tmp);
  }

Sample Code on how to Post Data to remote Web Page using HttpWebRequest

   Uri uri = new Uri("http://www.amazon.com/exec/obidos/search-handle-form/102-5194535-6807312");
   string data = "field-keywords=ASP.NET 2.0";
   if (uri.Scheme == Uri.UriSchemeHttp)
   {
       HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);
       request.Method = WebRequestMethods.Http.Post;
       request.ContentLength = data.Length;
       request.ContentType = "application/x-www-form-urlencoded";
       StreamWriter writer = new StreamWriter(request.GetRequestStream());
       writer.Write(data);
       writer.Close();
       HttpWebResponse response = (HttpWebResponse)request.GetResponse();
       StreamReader reader = new StreamReader(response.GetResponseStream());
       string tmp = reader.ReadToEnd();
       response.Close();
       Response.Write(tmp);
   }

Source

thanks!. this is helpful to begin with. – Bilal Fazlani Dec 22 '12 at 05:30 — Bilal Fazlani, Dec 22 '12 at 05:30

score 1 · Answer 3 · answered Dec 22 '12 at 05:05

1

Any HTTP client implementation, there are tons of open-source libraries for that. look at curl for example. Some dude made a .NET wrapper for it.

answered Dec 22 '12 at 05:05

littleadv

20,100
2
36
50

score 1 · Answer 4 · edited May 23 '17 at 11:53

You can continue using WebClient to POST (instead of GET, which is the HTTP verb you're currently using with DownloadString), but I think you'll find it easier to work with the (slightly) lower-level classes WebRequest and WebResponse.

There are two parts to this - the first is to post the login form, the second is recovering the "Set-cookie" header and sending that back to the server as "Cookie" along with your GET request. The server will use this cookie to identify you from now on (assuming it's using cookie-based authentication which I'm fairly confident it is as that page returns a Set-cookie header which includes "PHPSESSID").

Click Here to Check in Detail

How to interact with a website without a browser?

4 Answers4

Linked