Bash Script - Comparing last modified dates

Question

Trying to match the date output to the same format as curl last modified date which is: Last-Modified: Thu, 16 Apr 2020 08:14:26 GMT, with the Awk filter as in the script below it is 16 Apr 2020 09:27:51. The local file's last modified date will be compared with the remote file's last modified date. This script feels flaky and I am considering comparing date strings in a text file as opposed to relying on Http header feedback?

I am aware of the -z option in curl, but I want to give the user options via the if statements, it can't be fully automated update process, the user (admin) input is needed.

http header

HTTP/1.1 200 OK
Server: nginx
Date: Thu, 16 Apr 2020 08:43:13 GMT
Content-Length: 0
Connection: keep-alive
Last-Modified: Thu, 16 Apr 2020 08:14:26 GMT
Expires: Thu, 16 Apr 2020 08:44:13 GMT
Cache-Control: max-age=60
X-Cache-Status: MISS
X-Backend-ip: x.0.172.195
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Pragma: public
Cache-Control: public
Vary: Accept-Encoding
Accept-Ranges: bytes

script:

#!/bin/bash

local_file="/Users/usr/Desktop/file"
remote_file="www.someurl.com/file"


# current output format: 16 Apr 2020 08:14:26
remote_last_modified_date="$(curl -sI ${remote_file} | grep -E "Last-Modified:" | awk '{print $3,$4,$5,$6}' )"

# current output format: Thu Apr 9 18:15:30 SAST 2020  
local_last_modified_date="$(date -r "$local_file" )" 


parsed_remote_last_modified_date="$(date  +%s -d "$remote_last_modified_date")"  
parsed_local_last_modified_date="$(date -r "$local_file")"


  echo "local "$parsed_local_last_modified_date""
   echo "remote "$parsed_remote_last_modified_date""


if [ ${parsed_local_last_modified_date} -lt 
${parsed_remote_last_modified_date} ] ; then


    echo "A new version is available"

else

    echo "latest update already installed"
fi

error:

date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ... 
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]

Thanks for sharing code which you have tried, keep it up. Along with efforts sharing we encourage users to add samples of input and expected output too, so please do add the same in your question and let us know then. — RavinderSingh13, Apr 16 '20 at 11:27
did you try changing your date format `date '+%a, %d %b %T GMT'` it will return the `Thu, 16 Apr 2020 08:14:26 GMT` Last-Modified you said — A Khalili, Apr 16 '20 at 11:46
it gives the same error, I don't see syntax errors in the local_last_modified_date variable. tried the date -rf options - did not work either. — Adrian, Apr 16 '20 at 11:51
try `local_last_modified_date="$( date -r "${local_file}" '+%D %T' )` — Digvijay S, Apr 16 '20 at 12:17
I suggest you to get the timestamp corresponding to the last modification date instead of a complex string expressing the date in a sometimes ambiguous way. Further, the test operator `-ot` will fail because its operands must be filenames and not date strings like in your script. — Pierre François, Apr 16 '20 at 12:47
Note that `grep pattern|awk '{foo}'` is an anti-pattern, since awk should do the grepping: `awk '/pattern/ {foo}'`. You don't need `-E` for a fixed string, btw. — Jens, Apr 17 '20 at 10:33

Pierre François · Accepted Answer · 2020-04-17T18:47:07.493

1

I have rewritten the script without using the date command of the shell, which is not portable across MacOS and GNU Linux. I use ls instead, hoping this command is actually portable. Please check:

#!/bin/bash

## variables commented out since we don't have access to theirs values: 
#remote_file="www.someurl.com/file"
#remote_last_modified_date="$(curl -sI ${remote_file} | grep -E "Last-Modified:" | awk '{print $3,$4,$5,$6}' )"


## let's suppose we get next result into our variables:
local_file="test"
remote_last_modified_date="16 Apr 2020 08:14:26"

## we don't use the date command, not portable across MacOS and GNU Linux
## we use ls command with the option --time-style instead

## rewrite dates to allow comparison in format +"%Y-%m-%d_%H:%M:%S"
local_sortable_date=$(ls -l --time-style=+"%Y-%m-%d_%H:%M:%S" "$local_file"| cut -d ' ' -f 6)

remote_year=${remote_last_modified_date:7:4}
remote_month_name=${remote_last_modified_date:3:3}
remote_month=$(echo $remote_month_name | 
  sed 's/Jan/01/;s/Feb/02/;s/Mar/03/;s/Apr/04/;s/May/05/;s/Jun/06/;
    s/Aug/08/;s/Sep/09/;s/Oct/10/;s/Nov/11/;s/Dec/12/')
remote_day=${remote_last_modified_date:0:2}
remote_time=${remote_last_modified_date:12:8}

remote_sortable_date=${remote_year}-${remote_month}-${remote_day}_$remote_time

if [[ $remote_sortable_date > $local_sortable_date ]] ; then
  echo "A new version is available"
  set of commands
else
  echo "latest update already installed"
fi

edited Apr 17 '20 at 18:47

answered Apr 17 '20 at 15:49

Pierre François

5,850
1
17
38

thank you so much for all your input - it worked. the only thing i had to cgange was "ls" to "gls" after installing gnu core utilities. This works on ubuntu ? – Adrian Apr 17 '20 at 16:17
so mac is able to compare this format: 2020-04-16_08:14:26 , which I am very pleased about. – Adrian Apr 17 '20 at 16:22
The command `gls` doesn't exist on Linux Ubuntu, but you can define a kind of alias: `[ -x /bin/ls ] && LS=ls || LS=gls` and issue the command `$LS` instead of `ls` or `gls`, according to the presence or not of the executable file `/bin/ls`. – Pierre François Apr 17 '20 at 18:53
there is only one thing that I don't get. How do these numbers add up after the variables, i can see what it is doing (extracting), example remote_year=${remote_last_modified_date:7:4} ? – Adrian Apr 18 '20 at 07:07
This is the bash way to select a substring: `${var:7:4}` means the substring of `$var` starting at position `7` (the eighth character since the first postion is `0`) and `4` characters long. – Pierre François Apr 18 '20 at 08:55
You had me until *"I use `ls` instead."* Please [don't use `ls` in scripts.](https://mywiki.wooledge.org/ParsingLs) – tripleee Nov 03 '21 at 06:15
@tripleee: Indeed, this script should be adapted with `stat` and `sort` instead of `ls`. – Pierre François Nov 03 '21 at 11:22

Pierre François · Answer 2 · 2020-04-16T14:42:04.603

0

Try:

#!/bin/bash

## variables commented out since we don't have access to theirs values: 
#remote_file="www.someurl.com/file"
#remote_last_modified_date="$(curl -sI ${remote_file} | grep -E "Last-Modified:" | awk '{print $3,$4,$5,$6}' )"


## let's suppose we get next result into our variables:
local_file="test"
remote_last_modified_date="16 Apr 2020 08:14:26"

## let's compute timestamps
remote_last_modified_timestamp=$(date --date="$remote_last_modified_date" +"%s")
local_last_modified_timestamp=$(date +'%s' -r "${local_file}")

if [ $local_last_modified_timestamp -lt $remote_last_modified_timestamp ] ; then
  echo "A new version is available"
  set of commands
else
  echo "latest update already installed"
fi

edited Apr 16 '20 at 14:42

answered Apr 16 '20 at 13:13

Pierre François

5,850
1
17
38

thank you Pierre, i still get an illegal time format error. date: illegal option -- - usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ... [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format] date: illegal time format usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ... [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format] ./version_2.sh: line 18: [: : integer expression expected latest update already installed wolfs-MacBook:version-check wolf$ – Adrian Apr 16 '20 at 13:36
when i use the lt operator on my version, i get local LMD 01 Jan 1970 02:00:00 remote LMD 16 Apr 2020 08:14:26 ./version.sh: line 15: [: 01 Jan 1970 02:00:00: integer expression expected latest update already installed – Adrian Apr 16 '20 at 13:38
line 15: if [ "${local_last_modified_date}" -lt "${remote_last_modified_date}" ] ; then – Adrian Apr 16 '20 at 13:39
ignore the LMD part, that is just a test echo that i put in. – Adrian Apr 16 '20 at 13:40
I fear we don't have the same date function. On which system are you? I supppose MacOS. I am on Linux. The message `usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ... [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]` doesn't look like what I get on my system. – Pierre François Apr 16 '20 at 14:33
This: `if [ "${local_last_modified_date}" -lt "${remote_last_modified_date}" ]` will not work: you are comparing two strings where the shell is expecting two integers, i.e. two timestamps. I compare `$local_last_modified_timestamp` and `$remote_last_modified_timestamp`. Since these variables only contain integers, I will remove the double quotes from my code. – Pierre François Apr 16 '20 at 14:41
thank you @Pierre, also to no avail. will attempt it on Linux tomorrow, which distro you on? The ultimate goal for all my scripts is to work on both Mac and Linux, which is gonna be difficult with this one. Update 6 is the best result I got to this point. – Adrian Apr 16 '20 at 17:24
You are still trying to compare a date (character strings) and a timestamp (integer). This will not work. You have to compare timestamps with timestamps. Another possibility is to compare date strings as long as they are in a format where the year comes before the (zero padded) month, the month before the (zero padded) day, the day before the hour (zero padded and in the range 00-24), etc... – Pierre François Apr 17 '20 at 08:19
If you want your script to work on both Linux and Mac, you will have to avoid the use of the command date. – Pierre François Apr 17 '20 at 08:21
By the way, I am on Ubuntu Linux. I deduce that the script you are implementing has to run on all kinds of platforms. This is a good reason to avoid non portable commands as the shell *date* command. – Pierre François Apr 17 '20 at 09:28
"The -d flag does not work on OSX unless you download the GNU core utilities" – Adrian Apr 17 '20 at 09:43
local_file_1_stamp="$(date -r "${local_file}" +"%s" )" OUTPUT: 1586448930 – Adrian Apr 17 '20 at 10:58
remote_file_stamp="$(date -d "${remote_last_modified_date}" +"%s" )" OUTPUT = date format error – Adrian Apr 17 '20 at 11:01

Bash Script - Comparing last modified dates

2 Answers2