I want to crawl the data at the bottom of this webpage (the historical_table). If I use requests.get it returns the source code of the page, that contains only the script to create the table. The only way I know to crawl it is using selenium.webdriver (so via a browser session); however, this method is slower.
Is there a way to simulate the browser execution of the script without opening a browser session at all?
Asked
Active
Viewed 176 times
0
Matteo Felici
- 1,037
- 10
- 19
-
Possible duplicate of http://stackoverflow.com/questions/7593611/selenium-testing-without-browser check this – Vivek Kalyanarangan Mar 02 '17 at 08:23
-
No, I'd like to use another tool, not selenium. I'd like to use a requests-like process that gets the HTML of the page AND execute the script. – Matteo Felici Mar 02 '17 at 08:26
1 Answers
1
It is possible but it means you are writing your own browser which I dont recommend. Use a headless browser instead, like http://phantomjs.org . These are meant to be used this way.
Tamas Hegedus
- 28,755
- 12
- 63
- 97