how to change date-format in a log file using bash, avoiding while loop

Question

This is not a new question here and here, but the details make it differ.

My input log file looks like:

TEMP MON -=- Sat Aug 15 02:20:24 EEST 2020 -=- 48.6
TEMP MON -=- Sat Aug 15 02:20:50 EEST 2020 -=- 49.1
TEMP MON -=- Sat Aug 15 02:21:13 EEST 2020 -=- 49.1
TEMP MON -=- Sat Aug 15 02:21:44 EEST 2020 -=- 49.1
TEMP MON -=- Sat Aug 15 02:21:45 EEST 2020 -=- 48.6
TEMP MON -=- Sat Aug 15 02:21:52 EEST 2020 -=- 49.1
TEMP MON -=- Sat Aug 15 02:21:53 EEST 2020 -=- 48.6
TEMP MON -=- Sat Aug 15 02:21:54 EEST 2020 -=- 49.6
TEMP MON -=- Sat Aug 15 02:21:56 EEST 2020 -=- 49.1
TEMP MON -=- Sat Aug 15 02:21:57 EEST 2020 -=- 49.1

and the output should look like:

TEMP MON -=- 2020-08-15_02:20:24 EEST -=- 48.6
...

So it is simple enough to change the format of a date in bash using

date -d ${date_in_current_format} "+DATE_IN_NEW_FORMAT"

It is also possible (albeit inefficient) to iterate over the log file using a while loop and change the dates line by line (see the 1st link again).

However, I am looking for a bash solution that uses sed or perl (or awk or anything else for that matter) to carry out the same task.

The tip of what I have tried but still does not work are the following search and replace functions:

perl -pe "s/(.*) -=- (.*) -=- (.*)/\1 -=- $( date \2 "+%Z %Y-%m-%d_%H:%M:%S" ) -=- \3/" <file>

and with sed something similar:

sed "s:\(.*\) -=- \(.*\) -=- \(.*\):\1 -=- $( date -d \2 "+%Z %Y-%m-%d_%H:%M:%S" ) -=- \3:" <file>

In both cases the problem is that I cannot get the search and replace substitution "\2" to be expanded within the bash date command execution.

This might help: [Formatting date strings in a file with linux bash shell](https://stackoverflow.com/q/61015078/3776858) — Cyrus, Sep 02 '20 at 14:22
@anubhava: [Eastern European Summer Time](https://en.wikipedia.org/wiki/Eastern_European_Summer_Time) — Cyrus, Sep 02 '20 at 14:37
@Cyrus: Reason why I asked because I kept trying `TZ=EEST` in my `date` command and it wasn't working. After your answer I figured in terms of a Unix command it is `TZ=EET` instead. — anubhava, Sep 02 '20 at 14:57
In Perl, `$(` will be interpreted as a variable. And `\1` is better written as `$1`. — TLP, Sep 02 '20 at 16:28
@nass if you care for efficiency and have large log files to convert, calling `date` for every line would make any script extremely slow. — thanasisp, Sep 05 '20 at 02:43

thanasisp · Accepted Answer · 2020-09-02T18:59:46.367

With awk using only string functions, you can avoid calling the GNU awk datetime functions or the external command date, as we want to modify only the month and re-order the data.

> cat tst.awk
BEGIN { OFS=FS="-=-" }
{
    split($2, arr, " ")
    m=(index("JanFebMarAprMayJunJulAugSepOctNovDec", arr[2])+2)/3
    $2=sprintf(" %04d-%02d-%02d_%s %s ", arr[6], m, arr[3], arr[4], arr[5])
    print
}

Usage:

> awk -f tst.awk file
TEMP MON -=- 2020-08-15_02:20:24 EEST -=- 48.6
TEMP MON -=- 2020-08-15_02:20:50 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:13 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:44 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:45 EEST -=- 48.6
TEMP MON -=- 2020-08-15_02:21:52 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:53 EEST -=- 48.6
TEMP MON -=- 2020-08-15_02:21:54 EEST -=- 49.6
TEMP MON -=- 2020-08-15_02:21:56 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:57 EEST -=- 49.1

score 3 · Answer 2 · answered Sep 02 '20 at 14:54

3

You may use this awk solution:

awk 'BEGIN {
   FS=OFS=" -=- "
}
{
   cmd = sprintf("TZ=EET date -d \"%s\" +\"%Y-%m-%%d_%T %Z\"", $2);
   if ((cmd | getline output) > 0)
      $2 = output
   close(cmd)
} 1' file

TEMP MON -=- 2020-08-15_02:20:24 EEST -=- 48.6
TEMP MON -=- 2020-08-15_02:20:50 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:13 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:44 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:45 EEST -=- 48.6
TEMP MON -=- 2020-08-15_02:21:52 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:53 EEST -=- 48.6
TEMP MON -=- 2020-08-15_02:21:54 EEST -=- 49.6
TEMP MON -=- 2020-08-15_02:21:56 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:57 EEST -=- 49.1

answered Sep 02 '20 at 14:54

anubhava

761,203
64
569
643

1

The OP is clearly on Linux, so this is more for future visitors; this is the least portable of these answers, as it requires GNU `date`. – tripleee Sep 06 '20 at 17:49
`date -d` command is mentioned in OP's question itself. – anubhava Sep 07 '20 at 04:59
1

Yes yes, I just feel that it needed to be pointed out for people who are not necessarily on Linux. – tripleee Sep 07 '20 at 05:46

score 2 · Answer 3 · answered Sep 02 '20 at 17:58

Using core module Time::Piece in Perl:

perl -MTime::Piece -pe 's/-=-\s+\K(.*)(?=\s+-=-)/convert($1)/e;
  sub convert {
    $s = $_[0];
    $s =~ s/\s+EEST\s+/ /;
    $t = Time::Piece->strptime($s, "%a %b %d %T %Y");
    $res = $t->strftime("%Y-%m-%d_%H:%M:%S");
    "$res EEST"
 }' file

how to change date-format in a log file using bash, avoiding while loop

3 Answers3

Linked

Related