Selenium crontab script

Question

I have a simple Selenium Python script (I know this particular task can be done without Selenium, but it is just as an example)

from selenium import webdriver
driver = webdriver.Chrome("./lib/chromedriver")
options = webdriver.ChromeOptions()
options.add_argument("headless")
driver.get("https://www.xe.com/currencyconverter/convert/?Amount=1&From=ISK&To=" + currency.upper())
exchange_rate = driver.find_element_by_xpath('//*[@class="converterresult-toAmount"]').text
driver.close()
print("Exchange rate updated: ", exchange_rate)

when I run it simply in my terminal, as a user I see this output:

Exchange rate updated:  0.00724094

My crontab looks like this:

# m h  dom mon dow   command
SHELL=/bin/bash
PATH=/usr/local/bin/:/usr/bin:/usr/sbin

*/5 * * * *  DISPLAY=:0 python3 /path/to/my_script.py >  /path/to/logs.txt

I added DISPLAY=:0 per this answer

The log file is updated every 5 minutes, but is always empty, which leads me to believe that my Selenium script is not running correctly

EDIT: I made the following changes: to my_script:

options = webdriver.ChromeOptions()
options.add_argument("--no-sandbox")
options.add_argument("--headless")
options.add_argument("--disable-dev-shm-usage")
driver = webdriver.Chrome(PATH + "/lib/chromedriver", chrome_options=options)
driver.get("https://www.xe.com/currencyconverter/convert/?Amount=1&From=ISK&To=" + currency.upper())
exchange_rate = driver.find_element_by_xpath('//*[@class="converterresult-toAmount"]').text
driver.close()
print("Exchange rate updated: ", exchange_rate)

to the crontab:

SHELL=/bin/bash
PATH=/usr/local/bin/:/usr/bin:/usr/sbin
*/5 * * * *  DISPLAY=:0 /usr/bin/python3 /path/to/my_script.py > /path/to/logs.txt 2>&1

and now I have this error in the logs.txt file

Traceback (most recent call last):
  File "path/to/my_script.py", line 90, in <module>
    update_exchange_rate()
  File "path/to/my_script.py", line 69, in update_exchange_rate
    body = {"values": [[get_exchange_rate(currency=currency)]]}
  File "path/to/my_script.py", line 82, in get_exchange_rate
    driver = webdriver.Chrome("/path/to/lib/chromedriver", options=options)
  File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
    desired_capabilities=desired_capabilities)
  File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

Setting an environment variable in crontab is an awkward approach - add full path to python3 like `/usr/bin/python3`. — ipaleka, Aug 14 '19 at 15:31
@ipaleka I tried `*/5 * * * * DISPLAY=:0 /usr/bin/python3 /path/to/my_script.py > /path/to/logs.txt` but it still doesn't work :/ — Jessica Chambers, Aug 14 '19 at 15:51
Try with two dashes `--headless` and you may add `--disable-extensions` argument too. — ipaleka, Aug 15 '19 at 19:23
In script, `options.add_argument("--headless")` and another line `options.add_argument("--disable-extensions")`. — ipaleka, Aug 15 '19 at 20:30
@ipalekaok made all the changes (with and without dashes, to be sure) but still nothing — Jessica Chambers, Aug 15 '19 at 20:46

ipaleka · Answer 1 · 2019-08-16T09:33:26.457

1

I changed that currency.upper() to "USD" for testing purposes. Also changed your code as options should be passed to the driver during initialization.

from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--headless")
driver = webdriver.Chrome("/path/to/chromedriver", options=options)
driver.get("https://www.xe.com/currencyconverter/convert/?Amount=1&From=ISK&To=" + "USD")
exchange_rate = driver.find_element_by_xpath('//*[@class="converterresult-toAmount"]').text
driver.close()
print("Exchange rate updated: ", exchange_rate)

crontab:

*/5 * * * *  DISPLAY=:0 /usr/bin/python3 /path/to/my_script.py > /path/to/logs.txt 2>&1

The output:

$ cat /path/to/logs.txt
Exchange rate updated:  0.00806602

edited Aug 16 '19 at 09:33

answered Aug 15 '19 at 21:05

ipaleka

3,745
2
13
33

Somehow it still doesn't work - I've tried on two different computers too, with no success – Jessica Chambers Aug 16 '19 at 09:29
1

From where you import that `currency`? Use the changed cron line (added " 2>&1") so any error will be saved to log file. Post the error here. – ipaleka Aug 16 '19 at 09:35
the currency is stored in a global variable (and it works when I run it outside of cron) `CURRENCIES = ["eur", "usd", "gbp"]` -> `currency = CURRENCIES[0]` – Jessica Chambers Aug 16 '19 at 11:55
Tldr; I think something I'm doing wrong is causing an error here: `chrome/webdriver.py", line 81, in __init__ desired_capabilities=desired_capabilities)` – Jessica Chambers Aug 16 '19 at 12:27
1

Dunno, really, example code provided in this answer works flawlessly on my computer. – ipaleka Aug 17 '19 at 08:01

score 0 · Answer 2 · answered Jul 01 '20 at 06:30

For me after adding this options.binary_location = "/opt/google/chrome/chrome" as part of options object it worked

options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
options.add_argument("--no-sandbox")
options.add_argument('--disable-dev-shm-usage')
options.add_argument("--disable-setuid-sandbox")
options.add_argument('--remote-debugging-port=9222')
options.binary_location = "/opt/google/chrome/chrome"

Selenium crontab script

2 Answers2

Linked