0

I want to parse a website which contains a list of people and their information, The problem is that the website using ajax loads new and new information as I scroll down the website. I need information of ALL the people.

urllib.open(..).read() does not take care of the scroll down. Can you please suggest me a way to parse all the data.

Pranav Raj
  • 781
  • 1
  • 8
  • 19
  • 1
    Possible duplicate of http://stackoverflow.com/questions/8550114/can-scrapy-be-used-to-scrape-dynamic-content-from-websites-that-are-using-ajax and http://stackoverflow.com/questions/16390257/scrapping-ajax-pages-using-python – Girish May 23 '14 at 14:12

1 Answers1

1

You can use "Network" panel of chrome's devTool to figure out what is the path that the ajax requests to.

Then use python script to fetch the content with the path.

hoozecn
  • 497
  • 4
  • 14