Browserless web scraping of ajax page

Question

Have tried using Selenium after reading some tutorial for web scraping ..

The aim is to web/screen scrape a page that loads the required data after an Ajax call when (this ajax call made after Initial page load)..

The second aim is to run Selenium code in the background (not opening any browser) to allow loading the page (including the Ajax call) , retrieve the Final HTML and perform required processing locally ..

the code till now is as follows (code tutorial from http://www.geekonweb.com/net/web-page-scraping-using-selenium-and-net/)

public ActionResult Index()
    {
        //--
        //Below path should contain IEDriverServer.exe
        var chrome = new ChromeDriver(@"file path");
        chrome.Url = @"<url>";

        chrome.Navigate();

        //extract the html
        //var retval = ie.ExecuteScript("return document.body.outerHTML");

        string result = chrome.PageSource;


        return View();
    }

currently have not been able to find a way to run Selenium Silently (without GUI). kindly assist if that can be done.

Secondly kindly tell that how can Selenium be told to wait for the Ajax call to finish and then retrieve the data.

regards,

score 1 · Accepted Answer · edited May 23 '17 at 10:27

1

Here is a question on how to wait until an element is present. This is done to wait for the AJAX.

Here is a question on weather it's possible to run selenium headless.

edited May 23 '17 at 10:27

Community

1
1

answered Mar 16 '13 at 12:28

Benjamin Gruenbaum

270,886
87
504
504

the wait issue seems to have been somewhat solved :) . (somewhat) .. will try to look into headless running and inform in case it is achieved.. Thank you . – Abdul Ali Mar 18 '13 at 06:43
was Unable to Achieve headless browser issue.. However thank you for the answer . It seems to have solved half the problem. – Abdul Ali Mar 18 '13 at 12:47

Browserless web scraping of ajax page

1 Answers1

Linked