-1

How i can download webpage which uses java based loading mechanism? Code below returns nearly empty document due site mechanism. When viewed in browser you see "loading..." and after a while content is presented. Also i want to avoid using WebBrowser control.

HtmlDocument doc = new HtmlDocument();

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
            req.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;

if (!string.IsNullOrWhiteSpace(userAgent))
                req.UserAgent = userAgent;

if (cookies != null)
    {
        req.CookieContainer = new CookieContainer();

        foreach (Cookie c in cookies)
        req.CookieContainer.Add(c);
    }

var resp = req.GetResponse();
var resp_str = resp.GetResponseStream();
using (StreamReader sr = new StreamReader(resp_str, Encoding.GetEncoding("windows-1251")))
    {
        string r = sr.ReadToEnd();
        doc.LoadHtml(r);
    }

return doc;
Nerfpl
  • 167
  • 3
  • 8

1 Answers1

1

Well you basically need a web browser to do the javascript running. Your webrequest now only fetches the data, as is, from the server.

You could use System.Windows.Forms.WebBrowser but its not pretty. This https://stackoverflow.com/a/11394830/2940949 might give you some idea on the basic issue.

Community
  • 1
  • 1
Vasko Poposki
  • 132
  • 1
  • 7