0

guys. I'm newbie to Scrapy and learning how to work with it. An issue occurred I can't figure out what to do next, perhaps someone more experienced will help me: I have a basic web-site with a list of items I want to parse and download. The issue is that the page has filters - pressing will filter items out. Basically I want to download data after pressing one filter but can not figure out how to do that. I noticed that pressing this filter will not change page url but the page will be reloaded so this is not ajax. Filter is marked in HTML as a link with href="javascript:qsn.set('comm','0',1);". How can I imitate this filter press with Scrapy? Any help will be appreciated.

Nazarii Gudzovatyi
  • 195
  • 1
  • 4
  • 1
    examine and parse the html right off the bat. you dont have to imitate filter press. – taesu Jun 30 '15 at 20:05
  • Great advice, but the issue is that the items are not present in the list until the filter is pressed. – Nazarii Gudzovatyi Jun 30 '15 at 20:09
  • 1
    have you checked the html? is the item trivial? does clicking that filter fire xhr request? otherwise you don't really have to. it would help if you posted the url – taesu Jun 30 '15 at 20:09
  • Scrapy doesn't natively support javascript, so you will not be able to click on the link. The page is likely doing a javascript call to somewhere, and reloading the page with that data (that's AJAX, btw, even if it's not XML or asynchronous). Use a browser to see where the call is being made, and go directly to the source (see [here](http://stackoverflow.com/questions/8550114/can-scrapy-be-used-to-scrape-dynamic-content-from-websites-that-are-using-ajax)) – Rejected Jun 30 '15 at 20:14
  • Yes, clicking that filter fire XHR. There is a GET request to specific link with query parameters. Should I make request to that link? – Nazarii Gudzovatyi Jun 30 '15 at 20:39

0 Answers0