AWS not working working from Cronjob

Question

So I have a script to download a file from AWS daily and append it to a spread sheet. To do this I have set up a cronjob.

The Script works fine when I run it manually, but does not work when running from the cronjob.

The code has a line:

aws s3 cp s3://My/files/backup/ ~/home/AnPoc/ --recursive --exclude "*.tgz" --include "*results.tgz"

And in the email I recieve from the cronjob execution, I see the following error message:

./AnPoc/DayProcessing.sh: line 14: aws: command not found

I don't know why the command is not being found. Any help would be great.

This darn $hi7 should be documented by AWS in a section on creating cron for s3 cli. — Fr0zenFyr, Nov 23 '16 at 17:48

score 64 · Accepted Answer · answered Oct 21 '14 at 07:21

64

First: check where on your system the executable aws is stored. Use this command:

$ which aws
/usr/bin/aws # example output, can differ in your system

Now, place a variable called $PATH in your crontab before the script:

PATH=/usr/bin:/usr/local/bin

Those paths separated by : define where should be search for the exectable. In the example above it's /usr/bin. You have to check all executables in your cron job that they are available.

Another thing: try to avoid path with a tilde (~) in cronjobs. Use /home/user instead.

answered Oct 21 '14 at 07:21

chaos

8,162
3
33
40

4

If you `echo $PATH` you'll see that your session has the variable loaded. However when you execute from crontab it doesn't. – Ev. Sep 07 '16 at 09:01
1

Saved my ar$e... had to create and move backups of my mail server to s3 bucket. when I ran my script (with aws s3 cp ...) manually it worked, just adding the `PATH` was needed to make it work with cron. I swear if it were'nt for your post, I'd never be able to figure that out myself in another year. – Fr0zenFyr Nov 23 '16 at 17:42
1

@Ev. thanks for your comment, I noticed it even before the answer. It made me do a simple test by redirecting output of path to a file and the rest was explained in the answer above. – Fr0zenFyr Nov 23 '16 at 17:45
If you don't know, where to put in the crontab, see https://stackoverflow.com/a/2409369/1386952 – Christian Mar 18 '19 at 09:25
despite following all the requirements, still I am not able execute the cronjob when clubbing it with s3 command, can anyone please advice what am I doing wrong. `/usr/local/bin/aws s3 cp /home/ubuntu/sqlbackup.sql 's3://bucketName/db/sqlbackup.sql` – Sunil Kumar Feb 22 '22 at 11:39

score 36 · Answer 2 · edited Aug 18 '21 at 09:13

36

You should use the full path for the aws command. For example, /usr/local/bin/aws

edited Aug 18 '21 at 09:13

taras

6,566
10
39
50

answered Oct 21 '14 at 07:15

libregeek

1,264
11
23

1

@ user5735796 - when script is run through cron, it searches for aws executable in predefined locations as configured in $PATH variable. If aws is not installed in these locations, the command can't be found. So, either use the complete path to executable or add your aws location to $PATH variable. – plspl Mar 23 '17 at 03:15

score 10 · Answer 3 · answered May 31 '17 at 21:01

10

Put this code before your command line to be executed into crontab -e

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

answered May 31 '17 at 21:01

Roberto Carlos Reyes Fernandez

661
7
4

1

Thank you for your answer. It is very helpful for me to resolve a similar issue had with EKS commands such as `kubectl get po` – Damith Udayanga Nov 04 '21 at 13:38

score 3 · Answer 4 · answered May 29 '18 at 09:07

The only thing that worked for me was to specify the path explicitly in the script:

ROOTDIR=/home/myusername
LOGDIR=$ROOTDIR/logs
DUMPDIR=$ROOTDIR/db_backup
LOGFILE=$LOGDIR/db_backup.log

$ROOTDIR/.local/bin/aws s3 cp $DUMPDIR/myappname-`date +"%Y-%m-%d"` s3://my-bucket-name/backups/myappname-`date +"%Y-%m-%d"` --recursive >> $LOGFILE 2>&1

As a previous poster said, use which aws to find the location of aws.

score 0 · Answer 5 · answered Oct 29 '19 at 23:52

0

If you have aws in your profile, you could also include your profile by adding . $HOME/.profile

* * * * * . $HOME/.profile; /path/to/command

answered Oct 29 '19 at 23:52

Anthony Awuley

3,455
30
20

score 0 · Answer 6 · answered Aug 04 '22 at 13:13

If you install aws console using snap then the absolute/executable address will be like /snap/bin/aws otherwise its looks like /usr/bin/aws . It can be different according to your installation. You can find this path using $ whereis aws .

eg :

snap/bin/aws s3 sync --exclude "/path to exclude" /pathtosync s3://bucketname/

AWS not working working from Cronjob

6 Answers6

Linked