2

I need to run a cron job on a python script to generate basemap plots.

  1. The script by itself runs ok manually.

  2. A simple print("Hello") at the start of the program with the rest commented out also runs ok on cron with
    */10 * * * * /usr/bin/python3 ~/PythonFiles/TestScript.py > /dev/null 2>&1 >>log.txt

I made the file an executable using chmod +x and added a shebang (#!/home/usr/anaconda3/bin/python) at the start of the program. I can monitor activity in the log file via a printed message at the start of the program too

  1. When I come to run the "normal" program which includes modules (urllib.request, datetime, matplotlib, basemap, pygrib, numpy, ...), the script then stops outputting anything to log.txt

So I suspect it is to do with modules and possibly their locations. I checked and they seem to have been installed in various places (.../pkgs, .../conda-meta, .../site-packages, etc...)

First of all, is what I suspect correct?
Secondly, how do I fix it so that cron knows where to find all the libraries to run the job?

Many thanks!

Jérôme Teisseire
  • 1,518
  • 1
  • 16
  • 26
Stephane
  • 31
  • 7
  • 1
    Any error messages associated to that? Also, please include the relevant part of your script and the commands that you are using. Otherwise, it would be difficult tok now what is wrong – Andreas Nov 30 '18 at 01:39
  • Compare the output of `print(sys.path)` running from, `crontab` and `console`. – stovfl Nov 30 '18 at 19:19
  • @Andreas when I run tail -f /var/log/syslog, I get: myname CRON[4014]: (myname) CMD (/usr/bin/python3 ~/PythonFiles/TestScript.py > /dev/null 2>&1 >> log.txt) No error that I can see in there Script a collection of web data gathering and plotting. Too long to copy here. Imports: import urllib.request import datetime import matplotlib.pyplot as plt from mpl_toolkits.basemap import Basemap import pygrib import numpy as np from time import gmtime, strftime import matplotlib import metpy.calc as mcalc from metpy.units import units import calendar – Stephane Nov 30 '18 at 20:25
  • @stovfl Thanks for the link, I'll try to make sense of it. I'm still very new to the jargon – Stephane Nov 30 '18 at 20:38
  • I'm wrong, the Environment Variable reads [`PYTHONPATH`](https://stackoverflow.com/a/53098346/7414759), [how-to-get-cron-to-call-in-the-correct-paths](https://stackoverflow.com/a/2409369/7414759) – stovfl Nov 30 '18 at 20:43
  • @stovfl Running print(sys.path) in my python program after all the import commands returns: ['', '', '/home/stephane/anaconda3/lib/python36.zip', '/home/stephane/anaconda3/lib/python3.6', '/home/stephane/anaconda3/lib/python3.6/lib-dynload', '/home/stephane/anaconda3/lib/python3.6/site-packages', '/home/stephane/anaconda3/lib/python3.6/site-packages/IPython/extensions', '/home/stephane/.ipython'] – Stephane Nov 30 '18 at 21:12
  • @stovfl In the link you provided me with, there is a mention of "/etc/crontab must not be writable by any user other than root". What does it means in plainer English please? – Stephane Nov 30 '18 at 21:51
  • If, for example, numpy has the following path /home/stephane/anaconda3/pkgs, where is this supposed to go in my cron file? In PATH=...? And same for all my other imported modules. – Stephane Dec 01 '18 at 10:23
  • @Stephane: *"Running print(sys.path)"*: You didn't tell, if the output are running from **crontab**? The default file permission of `crontab` are `ls -l crontab >>>-rw-r--r-- 1 root root`. Use [`PYTHONPATH=`](https://stackoverflow.com/a/53098346/7414759) – stovfl Dec 01 '18 at 10:34
  • No output from crontab. I believe it is because sys.path requires import sys to be read from my program into crontab and this is the problem I'm facing to start with, i.e. crontab not getting the module locations. Back to previous question, If, for example, numpy has the following path /home/stephane/anaconda3/pkgs, where is this supposed to go in my cron file? In PATH=...? And same for all my other imported modules. – Stephane Dec 01 '18 at 10:53
  • And what do I do with the output of sys.path? Does it go as PYTHONPATH=... in crontab? – Stephane Dec 01 '18 at 11:07
  • To sum up so far PYTHON script: #!/home/stephane/anaconda3/bin/python print("Hello") CRON: SHELL=/bin/sh HOME=/home/stephane PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin * * * * * /usr/bin/python3 ~/PythonFiles/TestScript.py >/dev/null 2>&1 >> log.txt THAT WORKS. – Stephane Dec 01 '18 at 11:42
  • Running from distutils.sysconfig import get_python_lib; print get_python_lib() returns /home/stephane/anaconda3/lib/python3.6/site-packages. I believe this is where my libraries are located. How would that path be documented in cron? – Stephane Dec 01 '18 at 11:47

1 Answers1

1

I suspected it was to do with module location paths. After trawling through websites and tweaking inputs to cron, the following works!

SHELL=/bin/sh
HOME=/home/stephane
PYTHONPATH=/home/stephane/anaconda3/bin/python
PATH=/home/stephane/anaconda3/lib/python3.6/site-packages

*/2 * * * *  /home/stephane/anaconda3/bin/python ~/PythonFiles/TestScript.py >/dev/null 2>&1 >> log.txt

Note: matplotlib seems to need "import matplotlib as mpl; mpl.use('Agg')" to run off cron.

Thanks to all!

Stephane
  • 31
  • 7
  • I feel that `PYTHONPATH` and `PATH` should be reversed. The path to `bin/python` is required from the shell and should read `/home/stephane/anaconda3/bin` but it's not used as you give the full path to `bin/python`. The `PYTHONPATH` should hold **all** from your `sys.path` output. – stovfl Dec 03 '18 at 08:56