0

I have web page with strong ajax pagination (button for only next page). For go to page etc number 5, script should press button Next 5 times. But after script click - data for current page will lost.

It's possible return html content from Lua script to scrapy, and after this continue script run?

Now i use bad way. I merge html code for each page inside Lua script, and after last page i return it. But i think it's not good.

Verz1Lka
  • 406
  • 4
  • 15
  • Can you give the URL you are trying to crawl? Maybe it would be easier to manually send the requests for each page instead of "clicking" it. – rongon Jun 07 '17 at 07:03
  • https://www.moj.gov.sa/ar/opendata/bi/birealestate/Dashboards/300_kpiRegion/301_Monthly/kpi301_04_G.aspx But problem that site allow access only from Saud Arabia. – Verz1Lka Jun 07 '17 at 08:59
  • Yeah, I can't access it. I'd still suggest manually generating the requests, which IMO is clean and easy. This way you can scrape the data from one page, yield a request to the next and so on. See [here](https://stackoverflow.com/questions/8550114/can-scrapy-be-used-to-scrape-dynamic-content-from-websites-that-are-using-ajax) – rongon Jun 07 '17 at 09:11
  • It has realy big storng request. It's request more than firebug can show, it strip this. – Verz1Lka Jun 07 '17 at 09:29

0 Answers0