1

A simple script to list sites on my Webinoly Ubuntu 18 server works in the CLI but fails in cron. Webinoly is a site management script and has a command to list the sites it is managing:

site -list

The output from this command in the CLI looks like this:

 - catalyst.dk39ecbk3.com
 - siteexample3.com
 - webinoly.dk39ecbk3.com

The script I'm having trouble with (below) should remove the control characters, the " - " from the beginning of each line, and the blank lines using sed:

#!/bin/bash

# create array of sites using webinoly 'site' command
# webinoly site -list command returns lines that start with hyphens and spaces, along with control chars, which need to be removed
# SED s/[\x01-\x1F\x7F]//g removes control characters
# SED s/.{7}// removes first seven chars and s/.{5}$// removes last 5 chars
# SED /^\s*$/d removes blank lines
# removing characters http://www.theunixschool.com/2014/08/sed-examples-remove-delete-chars-from-line-file.html
# removing empty lines https://stackoverflow.com/questions/16414410/delete-empty-lines-using-sed

SITELIST=($(/usr/bin/site -list | sed -r "s/[\x01-\x1F\x7F]//g;s/.{7}//;s/.{5}$//;/^\s*$/d"))

#print site list

for SITE in ${SITELIST[@]}; do
echo "$SITE"
done

Here's the desired output, which I see in the CLI:

root@server1 ~/scripts # ./gdrive-backup-test.sh
catalyst.dk39ecbk3.com
siteexample3.com
webinoly.dk39ecbk3.com

The trouble happens when the script runs in cron. Here's the cron file:

root@server1 ~/scripts # crontab -l
SHELL=/bin/bash
MAILTO=myemail@gmail.com
15 3 * * 7 certbot renew --post-hook "service nginx restart"
47 01 * * * /root/scripts/gdrive-backup-test.sh > /root/scripts/output-gdrive-backup.txt

and here's the output-gdrive-backup.txt file generated by the cron command:

root@server1 ~/scripts # cat output-gdrive-backup.txt
lyst.dk39ecbk3
example3
noly.dk39ecbk3

The first three characters of each line are missing, as are the last four (the .com).

I've researched and made sure to force use of bash in the cron file as well as at the beginning of the script.

libbynotzoey
  • 481
  • 1
  • 4
  • 7
  • 1
    -> Could you explain why you need to remove the first 7 char and last 5 chars? What is the logic behind this? – Allan Dec 12 '18 at 07:25
  • This is the quantity I needed to remove to achieve the desired output in the CLI. I must not have been properly stripping the control characters. – libbynotzoey Dec 12 '18 at 17:45
  • What I am having trouble understanding is why the script works when run in CLI but not when run via cron. This is the biggest frustration of all. – libbynotzoey Dec 12 '18 at 17:45
  • 1
    Ok could you do the following 2 troubleshooting steps: (1) when the cron runs save the output of the site command in a file, we will need to check the output. (2) display the version of `sed` that the cron uses in another file – Allan Dec 13 '18 at 00:14
  • I found out that cron didn't have a TERM specified -- see answer. – libbynotzoey Dec 13 '18 at 20:07

2 Answers2

1

With the following input:

$ cat site 
 - catalyst.dk39ecbk3.com

 - siteexample3.com

 - webinoly.dk39ecbk3.com

 - webinoly.dk39ecbk3.com

you can use the following sed command to reach your output:

$ cat site | sed -e "s/^\s*-\s*//g;/^\s*$/d"
catalyst.dk39ecbk3.com
siteexample3.com
webinoly.dk39ecbk3.com
webinoly.dk39ecbk3.com

Replace the cat site by the command you want to filter the output from.

Allan
  • 12,117
  • 3
  • 27
  • 51
  • 1
    `s/[\x01-\x1F\x7F]//g;` eventually add this in the beginning of the `sed` if you have to remove those ctrl char – Allan Dec 12 '18 at 07:26
1

The answer turned out to be failure to specify TERM in my cron file. That solved the main issue I was having. This was a strange issue -- hard to research and figure out.

There were a few others -- one of them was that the path for one of the commands wasn't part of the path cron uses, but was included in the user root in the CLI. For more info on the TERM issue, see "tput: No value for $TERM and no -T specified " error logged by CRON process.

libbynotzoey
  • 481
  • 1
  • 4
  • 7