Why does selenium wait for a long time before executing this code?

Question

I'm trying to do infinite scrolling on this page and here is my code:

from selenium import webdriver
import time

profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override","Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0")
driver = webdriver.Firefox(profile)

driver.get("http://www.quora.com/Programming-Languages/followers")
for n in range(0,5): # For testing I have capped this at 5, will handle this properly once things start to work.
    driver.execute_script("window.scrollTo(0,1000000);")
    time.sleep(2)

So when I run this, it waits for a lot of seconds (more than 1 min sometimes) before doing any scrolling and then waits again for the same amount of time before the next scrolling. The code seems to work fine on other pages. Any ideas on how to fix this?

When I try to use Chrome instead of firefox, I get these errors: driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver') added to the .py file.

Traceback (most recent call last):
  File "ok.py", line 8, in <module>
    driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver')
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 65, in __init__
    keep_alive=True)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 73, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 121, in start_session
    'desiredCapabilities': desired_capabilities,
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 171, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 349, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 379, in _request
    self._conn.request(method, parsed_url.path, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 772, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 553, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

score 1 · Accepted Answer · edited May 23 '17 at 12:15

Switching to Chrome() helped me to solve the problem:

import time
from selenium import webdriver

followers_per_page = 18

driver = webdriver.Chrome()
driver.get("http://www.quora.com/Programming-Languages/followers")

# get the followers count
element = driver.find_element_by_class_name('count')
followers_count = int(element.text.replace('k', '000').replace('.', ''))
print followers_count

# scroll down the page iteratively with a delay
for _ in xrange(0, followers_count/followers_per_page + 1):
    driver.execute_script("window.scrollTo(0, 0,1000000);")
    time.sleep(2)

FYI, I'm, using a bit different approach: parsing the count of followers and calculating followers per page taking into account the fact that it loads 18 followers at a time.

I've actually worked on a similar quora question before, see:

Scrolling web page using selenium python webdriver

Well, this was not the first thing coming into my mind. Here's the story.

The problem is that there are pending requests to the http://tch840195.tch.quora.com/up/chan5-8886/updates URL that are taking minutes to complete. This is what makes selenium think the page is not completely loaded. And, things are getting worse - this is a periodical thing that happens every X seconds. Think about it as long pooling.

I've tried multiple things to overcome the problem using Firefox webdriver:

set webdriver.load.strategy preference to unstable
set network.http.response.timeout, network.http.connection-timeout and network.http.keep-alive.timeout and network.http.request.max-start-delay preferences
set page load timeout:
```
driver.set_page_load_timeout(3)
```
set script timeout:
```
driver.set_script_timeout(3)
```
call window.stop(); hoping it would stop active requests:
```
driver.execute_script('window.stop();')
```
updated to most recent Firefox and selenium package versions

One other option that might work is to somehow block the request to that "slow url" either using a proxy server and point firefox to it, or, if it is possible, let Firefox know to blacklist the URL (probably through an extension).

Also see the relevant issue with multiple workarounds inside:

Support a timeout argument on page load operations

Also see:

Thank you for looking at the problem. I tried to use chrome by doing this: `driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver')` but I get some errors. Updated in the question. — aste123, Oct 08 '14 at 14:29
@aste123 ok, looks like the path is configured correctly, do you have chrome browser installed? — alecxe, Oct 08 '14 at 14:40
Yes chrome is installed. `which` gives `/usr/bin/google-chrome` — aste123, Oct 08 '14 at 14:42
@aste123 ok, but what if you put path to chromedriver into the `PATH` env variable? Thanks. — alecxe, Oct 08 '14 at 14:45
I added it to PATH. `/home/asdf/apps/chromedrive/chromedriver` is shown in `echo $PATH`. But if I don't pass the path as an argument when creating the instance of `Chrome()`, I get an `not found in path` error. And if I pass it, I get the same error as I have mentioned above. Any idea what is wrong? — aste123, Oct 08 '14 at 14:56
@aste123 thanks, but what if you put `/home/asdf/apps/chromedrive/` to the PATH and use `driver = webdriver.Chrome()`? — alecxe, Oct 08 '14 at 14:57
It works the same way as passing the path as argument inside the code I guess. Same error as mentioned above. — aste123, Oct 08 '14 at 14:59

Why does selenium wait for a long time before executing this code?

1 Answers1