Screen Scraping a Javascript based webpage in Python

Question

I am working on a screen scraping tool in Python. But, as I look through the source of the webpage, I noticed that most of the data is coming through Javascript.

Any idea, how to scrape javascript based webpage ? Any tool in Python ?

Thanks

Duplicate of http://stackoverflow.com/questions/2148493/scrape-html-generated-by-javascript-with-python — hymloth, Nov 18 '11 at 14:16
Why you do consume the Javascript directly ? For instance how do you call the JS function `JS_Function(var1,var2,var3)` from python ? — Kiran, Nov 18 '11 at 21:34

score 5 · Accepted Answer · answered Nov 18 '11 at 14:09

5

Scraping javascript-based webpages is possible with selenium. In particular, try the Selenium WebDriver.

answered Nov 18 '11 at 14:09

unutbu

842,883
184
1,785
1,677

I tried Selenium. I donot want to mimic the user action. As I see it from running a sample program, it opens browser window and mimics the action. I donot want that. I want to extract the data from the webpage into my code. – Kiran Nov 19 '11 at 11:18
1

You don't have to mimic user actions if you don't need to. Just download the page and parse it. The point of using selenium is that it processes javascript for you. – unutbu Nov 19 '11 at 12:28

hoju · Answer 2 · 2012-09-24T00:58:31.667

4

I use webkit, which is the browser renderer behind Chrome and Safari. There are Python bindings to webkit through Qt.

And here is a full Python example to execute JavaScript and extract the final HTML.

edited Sep 24 '12 at 00:58

answered Dec 03 '11 at 03:28

hoju

28,392
37
134
178

score 3 · Answer 3 · answered Nov 18 '11 at 14:10

3

You can use the QtWebKit module of the PyQt4 library

answered Nov 18 '11 at 14:10

Will

4,498
1
21
20

Screen Scraping a Javascript based webpage in Python

3 Answers3

Linked

Related