2

I have a python script to take snapshots of the volumes of all running instances in AWS. The script is not working in crontab but it will will work when called otherwise.

I've written the output of the cronjob to a file and this is the output:

Traceback (most recent call last):
  File "completeBackUp2.py", line 2, in <module>
    import boto3
ImportError: No module named boto3

boto3 is installed and as previously mentioned, the script would work fine when called otherwise.

This is what my crontab looks like:

0 1 * * * /usr/bin/python  /opt/scripts/backup/completeBackUp2.py > /opt/scripts/backup/output 2>&1
Matthew Story
  • 3,573
  • 15
  • 26
nAJ
  • 33
  • 1
  • 8
  • 1
    cron explicitly unsets all environment variables as a matter of security. This includes PYTHONPATH. In order to help find out where boto3 lives: `python -c 'import boto3; print(boto3.__file__)'`. – Matthew Story Jul 07 '18 at 10:34

2 Answers2

1

The crontab is probably using a different version of Python as what you're using when running manually. Each installation of Python will load up its respective sys paths and they'll likely differ. I ran into this same error and fixed the issue by using the output of which python from where I was running my script manually as the command in the crontab. More info below.

When you're running your script manually via the command line, (e.g., python file_that_imports_boto3.py), you're using whatever version of Python is configured for that location (assuming there are multiple Python installations on your machine). If you run which python where you run the Python file manually you can see which Python installation is being used. In my case, the output of which python was 

~/miniconda3/bin/python

My crontab, however, was using 

/usr/lib64/python2.7

(which is standard location for Python 2.7 on Amazon EC2). So changing my crontab from 

0 1 * * * python file_that_imports_boto3.py

to

0 1 * * * ~/miniconda3/bin/python file_that_imports_boto3.py

resolved the issue for me. Hope this helps!

joshwa
  • 450
  • 3
  • 18
-1

Obviously the import PATH is different when you run the script from the command prompt and when it's run from cron.

Find out what is the import PATH:

$ python
>>> import sys
>>> sys.path

Then copy-paste the list of directories into the beginning of your script, like:

import sys
sys.path = [ ... what you found in the previous step ... ]

import boto3    # this should be AFTER you've made changes to sys.path
lenik
  • 23,228
  • 4
  • 34
  • 43
  • Thanks lenik. But still the same error on cron – nAJ Jul 07 '18 at 10:25
  • #!/usr/bin/python import boto3 import datetime import sys sys.path = ['/opt/scripts/backup', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/abhishek/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages'] – nAJ Jul 07 '18 at 10:28
  • @user382086 could you please find where your `boto` package is installed and just add that path to sys.path: `sys.path.append( BOTO_PATH )` ? you can easily do that by running python, and `import boto` + `boto.__path__` – lenik Jul 07 '18 at 10:30
  • Direct manipulation of sys.path should nearly always be avoided. setting PYTHONPATH is vastly preferable. – Matthew Story Jul 07 '18 at 10:31
  • When I'm doing import boto + boto.__path__, I'm getting the error Traceback (most recent call last): File "", line 1, in ImportError: No module named boto – nAJ Jul 07 '18 at 10:37
  • @user382086 `import boto3`, then `boto3.__path__', sorry I missed the `3` the first time. – lenik Jul 07 '18 at 10:38
  • @lenik Same error :( – nAJ Jul 07 '18 at 10:46
  • @nAJ do you by any chance `import boto3` BEFORE making changes to `sys.path` ? =) – lenik Jul 07 '18 at 12:05
  • I installed boto3 and then I added the script to cron. After that, I followed the above instructions as such in the same order – nAJ Jul 08 '18 at 03:55
  • @nAJ in your python script you have to put `import sys`, `sys.path = ...` BEFORE `import boto3`, it does not matter, when you installed it. check the script, reorder the lines. – lenik Jul 08 '18 at 04:58
  • No. I'll include my script here – nAJ Jul 08 '18 at 05:20
  • #!/usr/bin/python import datetime import sys sys.path = ['/home/abhishek/.local/lib/python2.7/site-packages/boto3'] import boto3 ec2 = boto3.resource('ec2') print("\n\nAWS snapshot backups starting at %s" % datetime.datetime.now()) instances = ec2.instances.filter( Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]) for instance in instances: instance_name = filter(lambda tag: tag['Key'] == 'Name', instance.tags)[0]['Value'] and it goes on – nAJ Jul 08 '18 at 05:21
  • After I've rearranged the lines, ie boto3 after import sys, the script is not working outside cron ie, the script is only working when boto3 is given BEFORE import sys – nAJ Jul 08 '18 at 05:29
  • please, change `sys.path = ...` into `sys.path.append( ... )` ? – lenik Jul 08 '18 at 05:30
  • another thing I'd like you to try is to append '/home/abhishek/.local/lib/python2.7/site-packages', without the 'boto3' at the end – lenik Jul 08 '18 at 05:32
  • "another thing I'd like you to try is to append '/home/abhishek/.local/lib/python2.7/site-packages', without the 'boto3' at the end " It worked with this. Thanks a lot @lenik – nAJ Jul 08 '18 at 05:42