How to parse html that includes javascript code

Question

How does one parse html documents which make heavy use of javascript? I know there are a few libraries in python which can parse static xml/html files and I'm basically looking for a programme or library (or even firefox plugin) which reads html+javascript, executes the javascript bit and outputs html code without javascript so it would look identical if displayed in a browser.

As a simple example

<a href="javascript:web_link(34, true);">link</a>

should be replaced by the appropriate value the javascript function returns, e.g.

<a href="http://www.example.com">link</a>

A more complex example would be a saved facebook html page which is littered with loads of javascript code.

Probably related to How to "execute" HTML+Javascript page with Node.js but do I really need Node.js and JSDOM? Also slightly related is Python library for rendering HTML and javascript but I'm not interested in rendering just the pure html output.

Either get a JavaScript runtime and sort something out with it, or analyse the code and work out what it's going to end up (strongly per-site configuration). — Chris Morgan, Aug 17 '11 at 14:46
http://stackoverflow.com/questions/19465510/how-to-parse-a-webpage-that-includes-javascript?rq=1 — gliptak, Oct 31 '13 at 01:30

score 3 · Answer 1 · answered Aug 22 '11 at 20:15

You can use Selenium with python as detailed here

Example:

import xmlrpclib

# Make an object to represent the XML-RPC server.
server_url = "http://localhost:8080/selenium-driver/RPC2"
app = xmlrpclib.ServerProxy(server_url)

# Bump timeout a little higher than the default 5 seconds
app.setTimeout(15)

import os
os.system('start run_firefox.bat')

print app.open('http://localhost:8080/AUT/000000A/http/www.amazon.com/')
print app.verifyTitle('Amazon.com: Welcome')
print app.verifySelected('url', 'All Products')
print app.select('url', 'Books')
print app.verifySelected('url', 'Books')
print app.verifyValue('field-keywords', '')
print app.type('field-keywords', 'Python Cookbook')
print app.clickAndWait('Go')
print app.verifyTitle('Amazon.com: Books Search Results: Python Cookbook')
print app.verifyTextPresent('Python Cookbook', '')
print app.verifyTextPresent('Alex Martellibot, David Ascher', '')
print app.testComplete()

score 2 · Answer 2 · edited Jun 20 '20 at 09:12

2

From Mozilla Gecko FAQ:

Q. Can you invoke the Gecko engine from a Unix shell script? Could you send it HTML and get back a web page that might be sent to the printer?

A. Not really supported; you can probably get something close to what you want by writing your own application using Gecko's embedding APIs, though. Note that it's currently not possible to print without a widget on the screen to render to.

Embedding Gecko in a program that outputs what you want may be way too heavy, but at least your output will be as good as it gets.

edited Jun 20 '20 at 09:12

Community

1
1

answered Aug 17 '11 at 08:27

Jonas G. Drange

8,749
2
27
38

Could also add this recipe: http://siliconforks.com/doc/parsing-javascript-with-spidermonkey/ – Jonas G. Drange Aug 31 '11 at 08:36

score 0 · Answer 3 · answered Oct 31 '13 at 01:34

0

PhantomJS can be loaded using Selenium

$ ipython

In [1]: from selenium import webdriver

In [2]: browser=webdriver.PhantomJS()

In [3]: browser.get('http://seleniumhq.org/')

In [4]: browser.title
Out[4]: u'Selenium - Web Browser Automation'

answered Oct 31 '13 at 01:34

gliptak

3,592
2
29
61

How to parse html that includes javascript code

3 Answers3

Linked