Scraping sites with javascript screen delay

Question

I'm attempting to scrape a site that has a split second javascript delay.

I'm currently using python for scraping. Whenever I 'get' the page, the javascript delay has not finished and is has not completely loaded the new dom yet.

How would I scrape such a pge?

Can you post the URL you want to scrape, or better yet a minimal example that reproduces the behaviour? — fmark, Feb 03 '11 at 08:07
Are you using urllib[2] to get the page, or are you using some browser? What is actually doing the dom load? — Gary van der Merwe, Feb 03 '11 at 09:22
I built SnapSearch for this purpose. https://snapsearch.io/ It was designed for SEO for JS, HTML5 and SPA apps. — CMCDragonkai, Apr 03 '14 at 06:22

score 1 · Answer 1 · answered Feb 11 '11 at 03:30

You can extend Mozilla to build a web scraper which can leverage the full power of the web browser. After all data have been loaded and the DOM has been built, you can extract needed data from the DOM using XSLT. If the DOM was dynamically changed after initial loading, you can take some approaches to wait for the changes. Visit http://www.gooseeker.com for more information. GooSeeker publish a similiar tool free for everyone. Most of codes are in javascript and readible, from which you can find how it runs.

Scraping sites with javascript screen delay

1 Answers1

Linked