0

I've written a bash script to automate month-end data processing from a vendor to a MySQL DB located on my machine.

The script runs to completion but quite commonly errs out/omits a few of the sections, which I'm subsequently able to run successfully via the command line without modification. The error appears to be in the 2nd cURL call (included below). It's not clear to me what about running within a shell script would cause this issue when I'm able to run without issue from the command line.

I've also included a call made just prior, and associated status updates, which completes without issue.

Sample code

DATE=$(date +"%Y%m")
USER=/user/
PASS=/pass/
SDATE=/lastRunDate/
MON=`date "+%m"`


#Indicators Inc/Bal Statement Descriptions
URL=`curl -s 'https://www.quandl.com/api/v3/datatables/SHARADAR/INDICATORS?qopts.export=true&api_key=APIKEY' | jq -r '.datatable_bulk_download.file.link'`
curl $URL -o /media/E/data/qu/INDICATORS.zip


#SEP Equity prices
URL=`curl -s 'https://www.quandl.com/api/v3/datatables/SHARADAR/SEP?qopts.export=true&api_key=APIKEYG&date.gt='$SDATE'' | jq -r '.datatable_bulk_download.file.link'`
curl $URL -o /media/E/data/qu/SEP.zip

unzip -p /media/E/data/qu/SEP.zip > /media/E/mysql/mysql-files/sep.csv
mysql --user=$USER --password=$PASS -e "load data infile '/media/E/mysql/mysql-files/sep.csv' into table eq.prices fields terminated by ',' optionally enclosed by '\"' lines terminated by '\n' ignore 1 lines (ticker,date,open,high,low,close,@v,dividends,unadjClose,@dummy) set volume = if(@v is NULL or @v = '' or @v = ' ', 0, @v);"
rm /media/E/mysql/mysql-files/sep.csv
mv /media/E/data/qu/SEP.zip /media/E/data/qu/archive/SEP$DATE.zip

Update/Error Messages

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13089  100 13089    0     0  32478      0 --:--:-- --:--:-- --:--:-- 32478
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: null
mysql: [Warning] Using a password on the command line interface can be insecure.
mv: cannot stat '/media/E/data/qu/SEP.zip': No such file or directory
Chris
  • 1,401
  • 4
  • 17
  • 28
  • 3
    why do you have "curl" in ```$URL```? aren't you essentially doing this: ```curl curl -s ...```? – ewokx Aug 17 '20 at 03:31
  • @ewong, the webpage presents a download link rather than the zipped CSV directly. I'm collecting that link in the first call – Chris Aug 17 '20 at 03:34
  • You need to add error checking and reporting to the script, in order to find out what's going on. I'd remove the `-s` flag from the `curl` calls that get URLs, and add the `-f` flag to *all* `curl` calls, and also add code after each `curl` to check its exit status to stop & report any error. My guess is that the `curl` to get the download link (that is, the second one in `URL=curl...` is failing, but I have no idea why without more info. – Gordon Davisson Aug 17 '20 at 04:47
  • @GordonDavisson, is `-v` output sufficient or are you looking for other outputs – Chris Aug 17 '20 at 05:30
  • @Chris `curl -v` is probably more detailed than you want for general use, but might be helpful while figuring out the specific problem here. I was thinking more along the lines of inspecting the exit code from `curl -f` to see what it told you (see my answer [here](https://stackoverflow.com/questions/62033740/bash-conditional-based-on-specific-exit-code-of-command/62033933#62033933) for some examples of how to do this -- although in your case, just printing the status code to stderr would be a good start). – Gordon Davisson Aug 17 '20 at 07:52

0 Answers0