Is this possible to load the page after the javascript execute using python?

Question

Here is the page I read:

<html>
<head>
<script type="text/javascript">
    document.write("Hello World")
</script>
</head>
<body>
</body>
</html>

As you can see, the Hello World is added on the HTML page using javascript, when I use the HTML parser, like the BeautifulSoup to parse it, it can't parse the Hello World, it is possible to me parse the actually result on how the client side really see....? Thanks.

It might be possible to parse this trivial case but otherwise you need to run all javascript on the page as a real javascript engine to see the result. — Esailija, Jun 15 '12 at 08:48
@Ted: You might want to take a look at this: http://stackoverflow.com/questions/6025082/headless-browser-for-python-javascript-support-required — , Jun 15 '12 at 09:02

score 17 · Answer 1 · edited Jan 18 '17 at 11:30

I ran into a similar problem when writing web scrapers in python, and I found Selenium Web Driver in combination with BeautifulSoup very useful. The code ends up looking something like this:

from selenium import webdriver
browser = webdriver.Firefox()
browser.get("http://www.yoursite.com")
soup = BeautifulSoup(browser.page_source, "html.parser")
...

With Selenium WebDriver, there's also functionally for a "wait until a certain DOM element has loaded", which makes the timing with javascript elements easier too.

score 3 · Answer 2 · answered Jun 15 '12 at 08:50

For a correct representation of what the DOM looks like after javascript manipulation, you'll have to actually execute the javascript. This has to be done by something that has a javascript engine and a DOM (rather than text/markup) representation of the document - typically, a browser.

Is this possible to load the page after the javascript execute using python?

2 Answers2

Linked

Related