2

I'm trying to crawl/scrape a website using c# windows app, getting the initial page is not a problem, but since data is displayed in a multi-page grid, I need to click "next" in code to retrieve more data.

This site's nextButton calls this javascript routine:

function __doPostBack(eventTarget, eventArgument) {
    if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
        theForm.__EVENTTARGET.value = eventTarget;
        theForm.__EVENTARGUMENT.value = eventArgument;
        theForm.submit();
    }
}

The eventTarget would be 'butNext' and eventArgument be ''.

With that said, I'm stuck on how to pass these parameters into WebRequest, or if I should use another API for doing this.

Thanks.

user1718553
  • 51
  • 1
  • 5
  • Assuming this is an aspx page this will be more complicate to replicate from c#. You need to pass the view state, cookies and some other hidden fields. All must be consistent or it won't work. What's the page you need to call? – derloopkat Jun 24 '16 at 22:24
  • it is an aspx page... Was afraid of that answer. – user1718553 Jun 24 '16 at 22:48
  • Are there other more manual methods on how I could call that or other javascript functions in Chrome or IE, but modify the parameters? Or better yet, can I not call the function within webrequest or webclient? To elaborate, I want to load a page which has a js function, and then i want to manually from within that browser trigger that function but pass other parameters into it. – user1718553 Jun 24 '16 at 22:50
  • If you are good in c# then pass cookies, the required hidden fields with the right encoding; that will work. Otherwise use Selenium or iMacros. These tools will run slower than C# because they open an actual browser and pull images, styles, etc. But the good thing is you don't need to care about cookies or hidden fields. – derloopkat Jun 24 '16 at 23:10
  • Possible duplicate of [Scrape ASP.NET website with paging](http://stackoverflow.com/questions/29727462/scrape-asp-net-website-with-paging) – derloopkat Jun 24 '16 at 23:18

0 Answers0