Execute Python (selenium) script in crontab

Question

I have read most of the python/cron here in stackoverflow and yet couldn't make my script run. I understood that I need to run my script through shell (using zsh & ipython by the way), but really I have no idea what to do :/

My simple code:

In the crontab-

*/1 * * * * ipython /home/usr/Data/progs/cron_test.py

My python script-

import pickle
    from selenium import webdriver

driver = webdriver.Firefox()
driver.get('http://www.google.com')

t=driver.current_url
pickle.dump(t,open('noreal','wb'))

I have tried some things already but to no avail:

#!python ../python etc
SHELL = /usr/bin/zsh
PATH =/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

Simply because I don't know what the problem is, I assume something with running python script via interpreter, but I don't know what I am doing really :)

A working solution would be great, and I would really appreciate an explanation to accompany any solution so I could understand the why and how and not just be like 'It works! Thanks! Bye!'

Thanks for any help!

Update So far I have narrowed down the problem, and now python runs with the following settings:

*/3 * * * * /usr/local/bin/ipython /home/user/Data/progs/RF/cron_test.py

and I got a traceback:

[1;31m---------------------------------------------------------------------------[0m
[1;31mWebDriverException[0m                        Traceback (most recent call last)
[1;32m/usr/local/lib/python2.7/dist-packages/IPython/utils/py3compat.pyc[0m in [0;36mexecfile[1;34m(fname, *where)[0m
[0;32m    176[0m             [1;32melse[0m[1;33m:[0m[1;33m[0m[0m
[0;32m    177[0m                 [0mfilename[0m [1;33m=[0m [0mfname[0m[1;33m[0m[0m
[1;32m--> 178[1;33m             [0m__builtin__[0m[1;33m.[0m[0mexecfile[0m[1;33m([0m[0mfilename[0m[1;33m,[0m [1;33m*[0m[0mwhere[0m[1;33m)[0m[1;33m[0m[0m
[0m
[1;32m/home/user/Data/progs/FB/cron_test.py[0m in [0;36m<module>[1;34m()[0m
[0;32m      9[0m [1;33m[0m[0m
[0;32m     10[0m [1;33m[0m[0m
[1;32m---> 11[1;33m [0mdriver[0m [1;33m=[0m [0mwebdriver[0m[1;33m.[0m[0mFirefox[0m[1;33m([0m[1;33m)[0m[1;33m[0m[0m
[0m[0;32m     12[0m [0mdriver[0m[1;33m.[0m[0mget[0m[1;33m([0m[1;34m'http://www.google.com'[0m[1;33m)[0m[1;33m[0m[0m
[0;32m     13[0m [1;33m[0m[0m

[1;32m/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.pyc[0m in [0;36m__init__[1;34m(self, firefox_profile, firefox_binary, timeout, capabilities, proxy)[0m
[0;32m     58[0m         RemoteWebDriver.__init__(self,
[0;32m     59[0m             command_executor=ExtensionConnection("127.0.0.1", self.profile,
[1;32m---> 60[1;33m             self.binary, timeout),
[0m[0;32m     61[0m             desired_capabilities=capabilities)
[0;32m     62[0m         [0mself[0m[1;33m.[0m[0m_is_remote[0m [1;33m=[0m [0mFalse[0m[1;33m[0m[0m

[1;32m/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.pyc[0m in [0;36m__init__[1;34m(self, host, firefox_profile, firefox_binary, timeout)[0m
[0;32m     45[0m         [0mself[0m[1;33m.[0m[0mprofile[0m[1;33m.[0m[0madd_extension[0m[1;33m([0m[1;33m)[0m[1;33m[0m[0m
[0;32m     46[0m [1;33m[0m[0m
[1;32m---> 47[1;33m         [0mself[0m[1;33m.[0m[0mbinary[0m[1;33m.[0m[0mlaunch_browser[0m[1;33m([0m[0mself[0m[1;33m.[0m[0mprofile[0m[1;33m)[0m[1;33m[0m[0m
[0m[0;32m     48[0m         [0m_URL[0m [1;33m=[0m [1;34m"http://%s:%d/hub"[0m [1;33m%[0m [1;33m([0m[0mHOST[0m[1;33m,[0m [0mPORT[0m[1;33m)[0m[1;33m[0m[0m
[0;32m     49[0m         RemoteConnection.__init__(

[1;32m/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.pyc[0m in [0;36mlaunch_browser[1;34m(self, profile)[0m
[0;32m     59[0m [1;33m[0m[0m
[0;32m     60[0m         [0mself[0m[1;33m.[0m[0m_start_from_profile_path[0m[1;33m([0m[0mself[0m[1;33m.[0m[0mprofile[0m[1;33m.[0m[0mpath[0m[1;33m)[0m[1;33m[0m[0m
[1;32m---> 61[1;33m         [0mself[0m[1;33m.[0m[0m_wait_until_connectable[0m[1;33m([0m[1;33m)[0m[1;33m[0m[0m
[0m[0;32m     62[0m [1;33m[0m[0m
[0;32m     63[0m     [1;32mdef[0m [0mkill[0m[1;33m([0m[0mself[0m[1;33m)[0m[1;33m:[0m[1;33m[0m[0m

[1;32m/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.pyc[0m in [0;36m_wait_until_connectable[1;34m(self)[0m
[0;32m     98[0m                 raise WebDriverException("The browser appears to have exited "
[0;32m     99[0m                       [1;34m"before we could connect. The output was: %s"[0m [1;33m%[0m[1;33m[0m[0m
[1;32m--> 100[1;33m                       self._get_firefox_output())
[0m[0;32m    101[0m             [1;32mif[0m [0mcount[0m [1;33m==[0m [1;36m30[0m[1;33m:[0m[1;33m[0m[0m
[0;32m    102[0m                 [0mself[0m[1;33m.[0m[0mkill[0m[1;33m([0m[1;33m)[0m[1;33m[0m[0m

[1;31mWebDriverException[0m: Message: 'The browser appears to have exited before we could connect. The output was: Error: no display specified\n'

What's the benefit of `ipython` for this epitome of *non*-interactive use? — tripleee, Dec 13 '13 at 20:59
Non i guess, sould it matter for the issue? I don't mind using plain Python — Captain_Meow_Meow, Dec 13 '13 at 21:29
What's the significance of the "things you tried already"? None of them are syntactically correct. A correct shebang would be something like `#!/usr/bin/env python` and the assignments must not contain any whitespace. — tripleee, Dec 14 '13 at 09:23
Thanks triplee i've changed the things. Now python works, and i got a traceback. — Captain_Meow_Meow, Dec 14 '13 at 13:11
The color codes are really disruptive, could you run it through `colcrt` to clean it up? — tripleee, Dec 14 '13 at 13:58
@Captain_Meow_Meow was a solution discovered? I am having the same issue with python, selenium and crontab — yoyoyoyo123, Apr 19 '18 at 18:01
I don't really remember :) I would suggest running in headless mode or you can try using Selenium docker container, you just need to point your test to use that. — Captain_Meow_Meow, Apr 21 '18 at 07:09

score 17 · Answer 1 · answered Jul 11 '14 at 22:28

17

As seen here, you need to give cron a display:

30 5 * * * export DISPLAY=:0; /home/me/good_morning.sh

answered Jul 11 '14 at 22:28

Gid

393
3
8

Adding display option solved my problem in ruby script. – Sanjay Prajapati Jan 25 '20 at 14:36
Worked for me using a Python script. – AtilioA Jul 29 '20 at 23:21

score 4 · Accepted Answer · edited Sep 22 '20 at 14:31

4

A quick guess, your job fails to run because it requires an X session. You should setup your test script to run in a headless mode.

update:
Your trace says exactly what I meant, firefox does not run without X, or a display if you want.

 The output was: Error: no display specified\n'

Save yourself more trouble, read the link I posted.

edited Sep 22 '20 at 14:31

e4c5

52,766
11
101
134

answered Dec 13 '13 at 21:10

oz123

27,559
27
125
187

I moved to something simpler just to check, did pickle.dump for a list into a file, doesn't create the file, so i take it as a proof that it's still not working. – Captain_Meow_Meow Dec 13 '13 at 21:26
did you see an error? does your script break? do other cronjobs work? – oz123 Dec 13 '13 at 21:41
a simple cronjob work:echo 'hello word' >> /tmp/hello.txt, in the python case i see nothing so i can't traceback the problem.. – Captain_Meow_Meow Dec 13 '13 at 21:52
open('cat_name_charlie_echo', 'wb')), you are opening a file, withou t a path. Try putting it with a full path where your cronuser can read and right and your user too. For example /tmp/cat_name.txt... – oz123 Dec 14 '13 at 09:02
Without any error message, there isn't much to go on. Does `python -v` run as you would expect from Cron? – tripleee Dec 14 '13 at 09:16
hi tripleee just addes the traceback i got, how would the python -v could help? – Captain_Meow_Meow Dec 14 '13 at 13:09
hi Oz123, it seems that running selenium headless would be the solution, but i do wish to run it with a GUI, so it's more of a last resort. could the problem be environment variables for Firefox path? – Captain_Meow_Meow Dec 14 '13 at 13:19
You cannot run a real GUI from Cron. You might be able to pull it off with something like xvfb but then that's probably what the headless does for you. – tripleee Dec 14 '13 at 14:00

Tomáš Marný · Answer 3 · 2018-06-06T13:28:33.600

You can do it without crontab.

I found schedule modul: GIThub/schedule

Now your code should look like:

import schedule
import time
def job():
    import pickle
    from selenium import webdriver
    driver = webdriver.Firefox()
    driver.get('http://www.google.com')
    t=driver.current_url
    pickle.dump(t,open('noreal','wb'))
schedule.every(1).minutes.do(job)
while True:
    schedule.run_pending()
    time.sleep(1)

It is not best solution but it works.

score 2 · Answer 4 · answered Aug 20 '20 at 19:55

This issue has bugged me for near 6 hours.

So if I write /usr/bin/python3 -u /root/apps/bridge/cinkselenium.py in terminal it is working fine.

But it doesn't work for crontab. Right?

Adding

from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()

will not help.

What you need is to make a shell script and write the following -

#!/usr/bin/env bash

export DISPLAY=:0
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
nohup /usr/bin/python3 -u /root/apps/bridge/cinkselenium.py > /root/apps/bridge/cinkselenium.log &y

Now run the shell script from Crontab -

* * * * * /root/apps/bridge/cinkselenium.sh

This is what works properly for selenium in crontab without headache in DigitalOcean servers. The paths will be different based on your server and program location.

score -1 · Answer 5 · answered Feb 15 '17 at 13:07

Install Xvfb or Xephyr or Xvnc. Also install pyvirtualdisplay python module. This is for Ubuntu 14.04 and tested on 16.04 as well.

sudo apt-get install python-pip
sudo apt-get install xvfb xserver-xephyr vnc4server
sudo pip install pyvirtualdisplay
# optional
sudo apt-get install python-pil scrot
sudo pip install pyscreenshot

Now you can start Xvfb from your python script. It will start Xvfb when your script will be executed. Add the following lines to your code.

from pyvirtualdisplay import Display
display = Display(visible=0, size=(1024, 768))
display.start()

Here is the example code.

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(1024, 768))
display.start()
browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()

display.stop()

Now you can mention the program in your crontab. But add the path of ipython instead of ipython only.

0 * * * * /usr/bin/python /home/usr/Data/progs/cron_test.py build

Execute Python (selenium) script in crontab

5 Answers5

Linked