Fast way of getting a page source

Question

I am currently using Selenium to open and retrieve the page source of a list of urls. However Selenium is taking way too long for each url and I plan on using this script for (atleast) a couple of hundred urls. Can anyone suggest a faster method of getting the page source given a url (using php maybe?)?

Please include the code of your suggestion. Thanks in advance.

browser.get(url)
body = browser.page_source

@BramVanroy does urllib2 give you access to the html code so that it can be parsed by beautiful soup? — Mohamad Moustafa, Jun 17 '19 at 14:06

score 0 · Answer 1 · answered Jun 17 '19 at 14:03

0

I'm a noob.

But I think requests might be faster, followed by headless browsers (Selenium, but doesn't open a GUI) and then finally, regular Selenium. I'm basing my thinking on the resources each method might use.

Unfortunately I can't find any articles that time the difference between these methods, but here's an article that seems to cover the chrome headless browser: https://intoli.com/blog/running-selenium-with-headless-chrome/

answered Jun 17 '19 at 14:03

Lafftar

145
3
11

I am not using GUI. I just want the source. – Mohamad Moustafa Jun 17 '19 at 14:04
You should use requests and see how much quicker that is. If you're using Selenium, you're likely using GUI. – Lafftar Jun 20 '19 at 01:11

Fast way of getting a page source

1 Answers1