I have an application where I need a long running instance of Selenium web driver (I am using Chrome driver 83.0.4103.39 in headless mode). Basically the app continuously pull url-data from a queue and gives the extracted url to Selenium which should perform some analysis on the website. Many of these websites could be down, unreachable or broken, so I've put a page load timeout of 10 seconds to avoid Selenium wait forever for page load.
The problem I am having here is that after some execution time (let's say 10 minutes) Selenium starts to give Timed out receiving message from renderer error for every url. Initially it works properly, it correctly opens the good websites and goes on timeout on the bad ones (website fails to load), but after some time it starts to give timeout on everything, even websites that should open correctly (I've checked, they open correctly on Chrome browser).
I am having hard time to debug this problem, since every exception in the application is caught correctly. I have also noticed that this problem happens only in headless mode.
- UPDATE *
During website analysis I also need to consider iframes (only top level), thus I've also added a logic to switch driver context to each iframe in the main page and extract the relative html.
This is a simplified version of the application:
import traceback
from time import sleep
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
width = 1024
height = 768
chrome_options = Options()
chrome_options.page_load_strategy = 'normal'
chrome_options.add_argument('--enable-automation')
chrome_options.add_argument('disable-infobars')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--lang=en')
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--allow-insecure-localhost')
chrome_options.add_argument('--allow-running-insecure-content')
chrome_options.add_argument('--disable-notifications')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument('--disable-browser-side-navigation')
chrome_options.add_argument('--mute-audio')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--force-device-scale-factor=1')
chrome_options.add_argument(f'window-size={width}x{height}')
chrome_options.add_experimental_option(
'prefs', {
'intl.accept_languages': 'en,en_US',
'download.prompt_for_download': False,
'download.default_directory': '/dev/null',
'automatic_downloads': 2,
'download_restrictions': 3,
'notifications': 2,
'media_stream': 2,
'media_stream_mic': 2,
'media_stream_camera': 2,
'durable_storage': 2,
}
)
driver = webdriver.Chrome(options=options)
driver.set_page_load_timeout(10) # Timeout 10 seconds
# Polling queue
while True:
url = queue.pop()
# Try open url
try:
driver.get(url)
except BaseException as e:
print(e)
print(traceback.format_exc())
continue
# Take website screenshot
png = driver.get_screenshot_as_png()
# Extract html from iframes (if any)
htmls = [driver.page_source]
iframes = driver.find_elements_by_xpath("//iframe")
for index, iframe in enumerate(iframes):
try:
driver.switch_to.frame(index)
htmls.append(driver.page_source)
driver.switch_to.default_content()
except BaseException as e:
print(e)
print(traceback.format_exc())
continue
# Do some analysis
for html in htmls:
# ...
pass
# Wait a bit
sleep(0.1)
This is an example of stack trace:
Opening https://www.yourmechanic.com/user/appointment/3732777/?access_token=HLZYIg&ukey=6quWpg1724633&rcode=abttgi&utm_medium=sms&utm_source= rb
LOAD EXCEPTION Message: timeout: Timed out receiving message from renderer: 10.000
(Session info: headless chrome=83.0.4103.116)
Traceback (most recent call last):
File "/Users/macbmacbookpro4ookpro4/Documents/Projects/python/proj001/main.py", line 202, in inference
driver.get(url)
File "/opt/anaconda3/envs/cv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "/opt/anaconda3/envs/cv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/opt/anaconda3/envs/cv/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 10.000
(Session info: headless chrome=83.0.4103.116)
Has anyone any clue why after a while of correct execution Selenium starts to give timeout exception for any url it tries to open?
