0

log.txt as below, the data is in a set contain ID, detection_time, Age and Height. My first part is to print the ID that appears again in the log.txt within 15 sec from its first appearance by using shell script which i've done it already. The second part is to print "detection_time", "Age" "Height" of the ID that match to my condition will also be printed with the ID too. The second part is where im stucked because i don't know how to develop the programming algorithm.

ID = 4231
detection_time = 1595556730 
Age = 25
Height = 182
ID = 3661
detection_time = 1595556737
Age = 24
Height = 182
ID = 2654
detection_time = 1595556740
Age = 22
Height = 184    
ID = 3661
detection_time = 1595556746
Age = 27
Height = 175
ID = 4231
detection_time = 1595556752
Age = 25

For example, log from above, the ID 3661 first appear at time 1595556737 and then appear again at 1595556746 which is just 9 sec after the first appearance. So it is matched to my condition which is want the ID that appear again within 15sec. After run a shell script, my desired output will be ID3661 with its latest detection_time Age and Height data of this ID 3661 which are

the matched ID is 3661
detection_time = 1595556746
Age = 27
Height = 175

Here is my code. I use associative array arr where id as key and detection_time as value. After run this script where above logfile as an input. The output will be The matched ID is 3661 without the ID detection_time, Age and Height which is where im stucked. Can anyone help me with this? Thank you.

#!/bin/bash
input="/tmp/log.txt"
declare -A arr
while read -r line
do

if [[ $line =~ ID ]] ; then
 id=$(echo $line | awk -F " " '{print $3}')
elif [[ $line =~ detection_time ]] ; then
 dtime=$(echo $line | awk -F " " '{print $3}')
 if  [[ arr["$id"] -ge $((dtime - 15)) ]]; then
  echo 'The matched ID is' "$id"
 fi
 arr["$id"]=$dtime
fi
done < "$input"
vgags
  • 3
  • 2
  • 1
    If have not understood completely what you want, but at least I can give you some hint with parsing a line: `if [[ $line =~ ^ID[[:space:]]=[[:space:]]([[:digit:]]+)$ ]]; then` this will put the `id` in `${BASH_REMATCH[1]}` and removes the need to use `awk` for this. – Marco Aug 29 '20 at 15:46
  • See also this earlier question by the same OP https://stackoverflow.com/questions/63228001/extract-the-unpredictable-data-that-have-its-own-timestamp-in-a-log-file-using-a – tripleee Aug 29 '20 at 19:39
  • 1
    Don't use a shell loop for this, see [why-is-using-a-shell-loop-to-process-text-considered-bad-practice](https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice) – Ed Morton Aug 30 '20 at 13:33

2 Answers2

3

This task fits to awk. Script below will print all cases where some id reappears in <= 15 seconds from its last detection.

file tst.awk :

$1=="ID" { if (p) print(rec); p=0; rec=""; id=$NF }
$1=="detection_time" { if ($NF - t[id] <= 15) p=1; t[id]=$NF }
{ rec = (rec? rec RS $0: $0) }
END { if (p) print(rec) }

Usage:

> awk -f tst.awk file
ID = 3661
detection_time = 1595556746
Age = 27
Height = 175
thanasisp
  • 5,855
  • 3
  • 14
  • 31
1

I am not sure if this is exactly what you expect as output.

#!/bin/bash
input="/tmp/log.txt"
declare -A arr
mode=""
while read -r line
do

if [[ $line =~ ^ID[[:space:]]=[[:space:]]([[:digit:]]+)$ ]] ; then
 id=${BASH_REMATCH[1]}
 [ -n "${mode}" ] && printf "\n"
elif [[ $line =~ ^(Age|Height)[[:space:]]=[[:space:]]([[:digit:]]+)$ ]] && [ -n "${mode}" ]; then
 printf "and %s is %s " "${BASH_REMATCH[1]}" "${BASH_REMATCH[2]}"
elif [[ $line =~ ^detection_time[[:space:]]=[[:space:]]([[:digit:]]+)$ ]] ; then
 dtime=${BASH_REMATCH[1]}
 if  [[ arr["$id"] -ge $((dtime - 15)) ]]; then
  printf 'The matched ID is %d at first time: %s ' "$id" "$( date -Iseconds -d @${arr[${id}]} )"
  mode="1"
 else
  mode=""
 fi
 arr["$id"]=$dtime
fi
done < "$input"

This prints the following from your input:

The matched ID is 3661 at first time: 2020-07-24T04:12:17+02:00 and Age is 27 and Height is 175

Hope this helps you to make it like you need it.

Marco
  • 824
  • 9
  • 13