How to deal with JavaScript when fetching http Response in C# using HttpWebRequest?

Question

I have a C# code (it's a Web Application, that's hosted on IIS) where I use HttpWebRequest to get HttpWebResponse. There I make request to any website & get the response as string, then I analysis the response string. But recently I get the response where JavaScript performs data fetching after page is loaded in browser.

I tried to debug this in firebug & saw that at bottom of response there's a JavaScript function that updates the dom elements after pageload. Is there any way that I could do the same in my C# code. I have searched on net about this found not solution till now.

Following is the code I am using:

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

        foreach (Cookie cook in response.Cookies)
        {
            Cookie cookie = new Cookie();
            cookie.Name = cook.Name;
            cookie.Value = cook.Value;
            cookie.Domain = cook.Domain;
            cookie.Expires = DateTime.Now.AddDays(10);
            cookieList.Add(cookie);
        }

        string postData = string.Format("username=" + txtUserID.Text + "&password=" + txtPwd.Text + "&url=https://example.com/&game=");
        byte[] postBytes = Encoding.UTF8.GetBytes(postData);
        HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://login.example.com/Login/authenticate");
        req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0";
        req.KeepAlive = true;
        req.AutomaticDecompression = DecompressionMethods.GZip;

        ////set the cookie
        req.CookieContainer = new CookieContainer();
        foreach (Cookie cook in cookieList)
        {
            Cookie cookie = new Cookie();
            cookie.Name = cook.Name;
            cookie.Value = cook.Value;
            cookie.Domain = cook.Domain;
            cookie.Expires = DateTime.Now.AddDays(10);
            req.CookieContainer.Add(cookie);
        }

        req.Headers.Add("Accept-Encoding", "gzip, deflate");
        req.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");//en-GB,en-US;q=0.8,en;q=0.6
        req.Method = "POST";
        req.Host = "login.example.com";
        req.Referer = "https://login.example.com/Login/logout";
        req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";

        req.ContentType = "application/x-www-form-urlencoded;";
        req.ContentLength = postBytes.Length;

        //getting the request stream and posting data
        StreamWriter requestwriter = new StreamWriter(req.GetRequestStream(), System.Text.Encoding.ASCII);
        requestwriter.Write(postData);
        requestwriter.Close();
        HttpWebResponse myHttpWebResponse = (HttpWebResponse)req.GetResponse();
        Stream responseStream = myHttpWebResponse.GetResponseStream();
        StreamReader myStreamReader = new StreamReader(responseStream, Encoding.ASCII);
        string responseString = myStreamReader.ReadToEnd();
        myStreamReader.Close();
        responseStream.Close();
        myHttpWebResponse.Close();

You need something that can function like a browser and actually execute javascript. HttpRequest is not capable of doing this. There are several libraries and toools that capable of doing that (for example: https://github.com/cefsharp/CefSharp/). If you don't want to use third party library - you may be able to utilize WebBrowser control for that (though won't recommend). — Evk, Oct 10 '17 at 12:04
Can github.com/cefsharp/CefSharp work in Web Application hosted on IIS? — user1400290, Oct 10 '17 at 13:21
There is "offscreen" renderer there, which does not depend on any user interface (such as winforms or wpf). This renderer can work in any type of application, including web application. — Evk, Oct 10 '17 at 13:22
Ok, I have seen the link, is there any code samples for using this library, the code looks to be in winForms mode. — user1400290, Oct 10 '17 at 13:25
Here is sample code for offscreen rendering: https://github.com/cefsharp/CefSharp/tree/master/CefSharp.OffScreen.Example They load page (including running all scripts) then make a screenshot of it, but you can easily adapt it to your needs. — Evk, Oct 10 '17 at 13:28
@Evk I downloaded code, & tried to run it but it has so many build issues when running it on VS2012 Express, also I could not identify how I would be able to use this code in my Web Application? — user1400290, Oct 12 '17 at 08:36
You don't need to build that solution (won't work in VS2012 anyway, it's 2017 year after all), just install cefsharp offscreen nuget package. See here for similar question: https://stackoverflow.com/q/35471261/5311735 — Evk, Oct 12 '17 at 09:57

score 0 · Accepted Answer · answered Oct 14 '17 at 05:26

I finally got the easy solution to my need. following is the link that I followed: Link to tutorial

Following is the code that'll get the results:

First you'll need to import following:

using System.Drawing;
using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;
using System.Text.RegularExpressions;
using System.IO;
using HtmlAgilityPack;

Now the code:

        var options = new PhantomJSOptions();
        var driver = new PhantomJSDriver(options);
        driver.Manage().Window.Size = new Size(1360, 728);
        var size = driver.Manage().Window.Size;

        driver.Navigate().GoToUrl("https://example.com/");
        string url = driver.Url;
        //the driver can now provide you with what you need (it will execute the script)
        //get the source of the page
        var source = driver.PageSource;
        //fully navigate the dom
        var pathElement1 = driver.FindElementByName("username");
        var pathElement2 = driver.FindElementByName("password");
        var pathElement3 = driver.FindElementByXPath("//button[@class='SubmitButton']");

        pathElement1.Clear();
        pathElement1.SendKeys("username");
        pathElement2.Clear();
        pathElement2.SendKeys("password");
        pathElement3.Click();

        //Now get the response after login button click
        source = driver.PageSource;

How to deal with JavaScript when fetching http Response in C# using HttpWebRequest?

1 Answers1