0

I am getting the following error when Celery tasks try to use Selenium;

[2020-06-12 23:23:04,228: WARNING/ForkPoolWorker-1] Message: unknown error: Chrome failed to start: exited abnormally.
  (chrome not reachable)
  (The process started from chrome location /opt/python/run/venv/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
[2020-06-12 23:23:04,230: ERROR/ForkPoolWorker-1] Task Update All Results[73583ec6-474a-4f20-8d88-d7be0a150642] raised unexpected: NameError("name 'sleep' is not defined",)
Traceback (most recent call last):
  File "/opt/python/bundle/42/app/fixtures/models.py", line 1035, in get_flash_result
    driver = webdriver.Chrome(options=opts)
  File "/opt/python/run/venv/local/lib/python3.6/dist-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
    desired_capabilities=desired_capabilities)
  File "/opt/python/run/venv/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/opt/python/run/venv/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/opt/python/run/venv/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/opt/python/run/venv/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally.
  (chrome not reachable)
  (The process started from chrome location /opt/python/run/venv/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

I have a Django app on an EC2 server via Elastic Beanstalk which runs periodic tasks using Celery however when webdriver is set I get the error.

The Celery task calls a class method on my model which contains the selenium code. The error doesn't occur if I run the same code via Django shell or call the method from the admin dashboard - just when the Celery task is running.

the error occurs after the following code;

from selenium import webdriver
from selenium.common.exceptions import TimeoutException, NoSuchElementException, ElementClickInterceptedException, WebDriverException
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.chrome.options import Options
...
opts = Options()
opts.headless = True
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")
opts.add_argument("--remote-debugging-port=9222")
assert opts.headless  # Operating in headless mode

driver = webdriver.Chrome(options=opts)

My Django app runs on a virtual environment - /opt/python/run/venv - and chromedriver and chrome are available to it; /opt/python/run/venv/bin/chromedriver & /opt/python/run/venv/bin/google-chrome --> /opt/google/chrome/google-chrome

versions;

(venv) [ec2-user@ip-172-31-12-230 ~]$ google-chrome --version && which google-chrome
Google Chrome 83.0.4103.97 
/opt/python/run/venv/bin/google-chrome
(venv) [ec2-user@ip-172-31-12-230 ~]$ chromedriver --version && which chromedriver
ChromeDriver 83.0.4103.39 (ccbf011cb2d2b19b506d844400483861342c20cd-refs/branch-heads/4103@{#416})
/opt/python/run/venv/bin/chromedriver

(venv) [ec2-user@ip-172-31-12-230 ~]$ pip list
Package               Version
--------------------- ----------
celery                4.4.2
Django                3.0.4
django-celery-beat    2.0.0
selenium              3.141.0

EDIT This was linked to another question here and closed, however the link doesn't answer my question as I'm still getting the error. I tried making the following adjustments and validations;

  1. Google Chrome and Chromedriver are up to date
  2. executable path (/opt/python/run/venv/bin/chromedriver) to chromedriver was added to webdriver()
  3. .add_argument('--headless') was moved to be the first option set
  4. google-chrome in venv and /usr/bin are symlinks
  5. binary location option added
  6. tried deleting chrome and chromedriver and reinstalling (a few times)
  7. tried changing options to just '--no-sandbox' and '--disable-dev-shm-usage'
  8. I've tried putting it in a loop and adding a sleep
  9. Celery is not using root user or sudo

I only encounter this error when Celery worker executes tasks, I can run the same code being run by the worker in the django shell or via the admin dashboard and don't receive the error so it feels like this is a celery error and not my selenium options.

seanpconkie
  • 71
  • 1
  • 6

1 Answers1

0

Your code is working fine. There may be environment issue or chrome installation issue, re-installation of chrome may help. Further I have modified code and added the default directory for chrome , check if it can work. 

from selenium import webdriver
from selenium.common.exceptions import TimeoutException, NoSuchElementException, ElementClickInterceptedException, WebDriverException
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.chrome.options import Options
import getpass
system_user = getpass.getuser()

opts = Options()
# opts.headless = True
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")
# opts.add_argument("--remote-debugging-port=9222")

# Adding only below line, modify the path of driver
opts.add_argument("user-data-dir=C:\\Users\\" + system_user + "\\AppData\\Local\\Google\\Chrome\\User Data")
driver = webdriver.Chrome(executable_path="C:\\chrome driver\\chromedriver.exe",options=opts)
driver.get("https://stackoverflow.com/")
Hietsh Kumar
  • 1,197
  • 9
  • 17