how to download a dynamically generated web page html in python

Asked Nov 08 '17 at 13:07

Active Oct 15 '18 at 15:40

Viewed 170 times

I am new to web service. Say I have a page url like below: http://xxx/xx/post?id=9898

If I use urllib.request.urlopen() method to download this url's html source, the html I get is different from what I see when open this page in a browser.

The difference is, when I open it in browser, I guess some JS code will run (with the given url parameter) and dynamically fetch and generate a lot of contents. But when I do urlopen(), basically this fetching and generating process is not triggered.

I am not sure if the term dynamic/static is correct to use here. Correct me if I am wrong, so urlopen() can only download static html page, if it is dynamic page, it will not work?

Can anyone show me how to download the html of fully generated version of a web page?

edited Oct 15 '18 at 15:40

asked Nov 08 '17 at 13:07

Acton

Use a headless browser. – bruno desthuilliers Nov 08 '17 at 13:09
You need access to an engine that will handle the JavaScript. Python by itself can't do that, but there are libraries that can, eg Selenium. – PM 2Ring Nov 08 '17 at 13:10
I would go for selenium in your case, here is a simulair question with solutions using selinium: https://stackoverflow.com/questions/15122864/selenium-wait-until-document-is-ready – Merijndk Nov 08 '17 at 13:12

how to download a dynamically generated web page html in python

0 Answers0