3

I am trying to scrape a website using Selenium and Chrome's webdriver and this all worked fine until I switched to a newer Macbook. All of sudden, the webdriver seems to not recognize when the website is actually fully loaded.

Error message goes as follows

TimeoutException: Message: timeout: cannot determine loading status from timeout: Timed out receiving message from renderer: -0.003
(Session info: chrome=54.0.2840.87) (Driver info: chromedriver=2.25.426935 (820a95b0b81d33e42712f9198c215f703412e1a1),platform=Mac OS X 10.12.1 x86_64)

My code looks like this:

      import os
      import time
      from selenium import webdriver
      driver = webdriver.Chrome(os.path.join(os.getcwd(), 'chromedriver'))
      driver.get('http://www.clever-tanken.de/')
Vini.g.fer
  • 11,639
  • 16
  • 61
  • 90
Haluka Maier-Borst
  • 75
  • 1
  • 4
  • 13

3 Answers3

7

This is a known bug in the chrome webdriver. Ex1, Ex2

After a very brief read, it looks like their developers are having some trouble reproducing the bug. @dimkin mentioned CDNs as a possible cause for the bug. It looks like others have reported the bug and have similar suspicions that cdn content is not resolving DNS appropriately.

I was able to isolate the issue. In my case it was happening only for pages referencing a static file (an image in a HTML tag for instance http://cdn.local.myproject.net/static/myimage.png) on my custom local cdn domain. The was not present if I used a relative path "/static/myimage.png" or localhost "http://127.0.0.1/static/myimage.png" so I figured it was a DNS problem.

I was able to bypass the problem by using the --dns-prefetch-disable option of chrome.

Example: 

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--dns-prefetch-disable")
driver = webdriver.Chrome(chrome_options=chrome_options)

Another workaround mentioned looks promising. It doesn't fix anything but could possibly get around the problem. It catches the exception and simply attempts a refresh:

You'll need: from selenium.common.exceptions import TimeoutException

try:
   webDriver.get(url);
except TimeoutException as ex:
   print(ex.Message)
   webDriver.navigate().refresh()
  • 1
    A question regarding the code in the yellow boy: where do you define the options in the six line? – Haluka Maier-Borst Nov 10 '16 at 16:54
  • Sorry, that came from a direct cut and paste from the bug report. I've corrected the code sample and it should run now. –  Nov 10 '16 at 18:51
  • Ok, thanks. Just tried the upper one out but still I get the timeout errors. About the second code: Where do I have to insert this? Looks like Javascript bc of the curly brackets....Sorry for being such a beginner. – Haluka Maier-Borst Nov 11 '16 at 11:09
1

So as strange as it might sound, it had to do with the language preferences. After I figured out that the last remaining difference between my old mac(where the code worked fine) and my new mac(on which the code kept crashing) was the language preferences, I changed it to English. Now, the code runs fine!

Haluka Maier-Borst
  • 75
  • 1
  • 4
  • 13
  • Glad you fixed it! It doesn't sound all that strange really, and it lines up with the whole "something in the background not being able to render appropriately" described in the bug reports. –  Nov 14 '16 at 14:41
0

this indicates some network issues. Faced same, when from one host our CDN wasn't accessible and some files failed to load Chrome/chromedriver/selenium versions manipulations doesn't helped. It was resolved only with fixes for network

dimkin
  • 108
  • 1
  • 11
  • Thank you for the answer but could you be a little more specific about the "network issues"? It is kind of hard to find a fix, if you just say it is about the network... – Haluka Maier-Borst Nov 10 '16 at 09:11
  • we running tests from dockers. route from some of theme wer closed in fw. So host just timed out. Check it with some devtools/fiddler or something like – dimkin Nov 10 '16 at 10:38