I am trying to scrape few webpages, and am using webdriver from Selenium to do so. I have noticed few pages are taking very long time to load, and am wondering what can I do to improve this behaviour.
I use BeautifulSoup to parse the content, which works fine. So, looking at ways to make the load faster. (I do not need images or css to load if that takes time to load, I tried to add service_args=['--load-images=no'], but that didn't help the case as well. )
My script has this:
scraper = webdriver.PhantomJS()
scraper.driver.get(URL) # this takes time
Also, I have noticed that when I open terminal and run these commands:
python
scraper = webdriver.PhantomJS()
scraper.driver.get(URL) # this takes much less time
I wonder why it runs so much quicker here than when running the same commands through a python file.
To brief out:
Solution to load pages faster
Why do much difference in running same lines of code in these scenerios
Thanks!
