I'm using scrapy to run tests using Selenium Webdriver. Most of the time script works fine, but sometimes it returns a Timeout error when initializing SeleniumMiddleware.
Below is my custom download middleware which inherits from SeleniumMiddleware.
class CustomDownloadMiddleware(SeleniumMiddleware):
def __init__(self, *args, **kwargs):
try:
super().__init__(*args, **kwargs) # This is where Timeout exception happens
except Exception as e:
print("-- Unable initialize Selenium Middleware ---")
print(e) # This is where it prints the exception message
def process_request(self, request, spider):
"""Process a request using the selenium driver if applicable"""
...
Below is the error message I'm receiving.
Message: session not created
source-app-1 | from timeout: Timed out receiving message from renderer: 600.000
source-app-1 | (Session info: headless chrome=102.0.5005.182)
I'm using Python with scrapy, scrapy_selenium libraries and I use Chrome webdriver. I see this failure at a rate of about 1 in 10.
FYI... There is a similar post related to this issue but there is no clear answer to what's causing it or how to fix it. (I've tried all posible solutions mentioned in that post) Selenium Timed out receiving message from renderer
UPDATE
I dive deeper and the Timeout exception is happening when the code initiates the webdriver.Chrome() class, from python's selenium packages.