1

I made a bash script that downloads a file from an amazon s3 bucket and then upload it back with some transformations on it. It works fine manually but as soon as I put it on the crontab I either have no file that is downloaded or an empty file. I get this error :

curl: (56) Received HTTP code 407 from proxy after CONNECT

I am using this code for my process :

#!/bin/sh 
outputFile="PATH"
amzFile="AMAZON_FILE_PATH"
bucket="BUCKET"
resource="/${bucket}/${amzFile}"
contentType="application/x-gzip"
dateValue=`date -R`
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="S3_KEY"
s3Secret="S3SECRET"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`

curl  -H "Host: ${bucket}.s3.amazonaws.com" \
     -H "Date: ${dateValue}" \
     -H "Content-Type: ${contentType}" \
     -H "Authorization: AWS ${s3Key}:${signature}" \
     https://${bucket}.s3.amazonaws.com/${amzFile} -o $outputFile

Any of you guys has an idea ?

Ali.D.Rinas
  • 11
  • 1
  • Are you running it from an EC2? – Shahad Ishraq Jul 08 '21 at 14:28
  • Maybe try to put the full path for openssl (and curl while at it). Path is different when using crontab than when logged in. – Andre Gelinas Jul 08 '21 at 14:52
  • @ShahadIshraq, no I am using a remote server of my company. – Ali.D.Rinas Jul 08 '21 at 15:33
  • @AndreGelinas I have already tried to put the paths of all the files and to put the paths 'bins/...' of the aws commands that I use but without success – Ali.D.Rinas Jul 08 '21 at 15:35
  • How do you authenticate to your corp proxy outside of crontab? Are you using `su` to run this crontab script as you (or is it running as root)? – jarmod Jul 08 '21 at 15:56
  • @jarmod I use aws configure and then I give my credentials region and output format. I am not root and calling crontab with 'crontab -e' or 'su Ali.D.Rinas crontab -e' doesn't change anything – Ali.D.Rinas Jul 08 '21 at 16:03
  • Those are AWS credentials, not proxy credentials. The 407 error seems to be from your corp proxy. I would assume that your client machine is pre-configured with the hostname of, and credentials for, the proxy. Do you have `HTTPS_PROXY` or `http_proxy` in your [environment](https://stackoverflow.com/questions/9445489/performing-http-requests-with-curl-using-proxy), for example? – jarmod Jul 08 '21 at 16:14
  • After having check, HTTPS_PROXY in our environment. I will give it a try with this and give feedbacks. – Ali.D.Rinas Jul 12 '21 at 13:52
  • Hello guys, sorry for my late response. After having put http_proxy=http://squid:XXXX and https_proxy=http://squid:XXXX (squid:XXXX manages the authorization for the server input and output), the download is going normally ! – Ali.D.Rinas Aug 16 '21 at 08:54

0 Answers0