it can only print the first and last value when using for loop to read an array in Bash

Question

There are some rotate gz log file in the log dir,it rotaes every twenty minutes using logrotate with dateformat '.%s'， as flows

ls -l /var/log/app/h323server.log.[1-9][0-9]*  |head
-rw-r--r-- 1 root adm   2063852 Mar 19 02:00 /var/log/app/h323server.log.1584554401.gz
-rw-r--r-- 1 root adm   2093937 Mar 19 02:20 /var/log/app/h323server.log.1584555601.gz

I want to print the corresponding log content between start_time timestamp and end_time timestamp，there are a few steps:

1, find out the log file and fill them into an array named totalfile

2, use for loop to read totalfile and print, the first and last element need to filter by start and end timestamp, print the rest of file direcly. I want to use for (( i=1; i<${arraylength}+1; i++ )); loop to achieve it，but something goes wrong.

The Bash script is as fllow:

#!/bin/bash

oldifs="$IFS"
IFS=$'\n'
declare -a filetime
declare -a filename
declare -a totalfile
index_1=0
index_2=0

for line in $(ls -l /var/log/app/h323server.log.[1-9][0-9]* |awk '{split($NF,a,".");print a[3],$NF}')
do
        filetime[${index_1}]=$(echo ${line} |awk '{print $1}')
        filename[${index_2}]=$(echo ${line} |awk '{print $2}')
        ((index_1++))
        ((index_2++))
done
IFS="$oldifs" 

index=0
timesys_s=1584945601
timesys_e=1584948001

# store the corresponding delaycompress and compress file to totalfile array
while [ ${index} -le $((${#filetime[@]}-1)) ]
do
        if [ ${index} -eq 0 ]
        then
                if [[ ${filetime[${index}]} -ge ${timesys_s} ]] || \
                   [[ ${filetime[${index}]} -le ${timesys_s} ]] || \
                   [[ (${filetime[${index}-1]} -ge ${timesys_s}) && (${filetime[${index}]} -le ${timesys_e}) ]]
                then
                        totalfile[${index}]=${filename[${index}]}
                fi
        else
                if [[ (${filetime[${index}-1]} -le ${timesys_s}) && (${filetime[${index}]} -ge ${timesys_s}) ]] || \
                   [[ (${filetime[${index}-1]} -ge ${timesys_s}) && (${filetime[${index}]} -le ${timesys_e}) ]] || \
                   [[ (${filetime[${index}-1]} -le ${timesys_e}) && (${filetime[${index}]} -ge ${timesys_e}) ]]
                then
                        totalfile[${index}]=${filename[${index}]}
                fi
        fi
        ((index++))
done

echo "length of totalfile:"
echo ${#totalfile[@]}
echo "content of totalfile:"
echo ${totalfile[@]}

# get length of totalfile
arraylength=${#totalfile[@]}

# use for loop to read all values and indexes
for (( i=1; i<${arraylength}+1; i++ ));
do
  echo $i " / " ${arraylength} " : " ${totalfile[$i-1]}
done

# can only print the first and last value when using ${array[index]} to loop
echo "the length of totalfile is: ${arraylength}"
echo "the 1st element: ${totalfile[0]}"
echo "the 2st element: ${totalfile[1]}"
echo "the 3st element: ${totalfile[2]}"
echo "the 4st element: ${totalfile[3]}"
echo "the 5st element: ${totalfile[-1]}"

the output is as follows:

length of totalfile:
5
content of totalfile:
/var/log/app/h323server.log.1584554401.gz /var/log/app/h323server.log.1584945601.gz /var/log/app/h323server.log.1584946801.gz /var/log/app/h323server.log.1584948001.gz /var/log/app/h323server.log.1584949201.gz
1  /  5  :  /var/log/app/h323server.log.1584554401.gz
2  /  5  : 
3  /  5  : 
4  /  5  : 
5  /  5  : 
the length of totalfile is: 5
the 1st element: /var/log/app/h323server.log.1584554401.gz
the 2st element: 
the 3st element: 
the 4st element: 
the 5st element: /var/log/app/h323server.log.1584949201.gz

The question is:

There are five element in the totalfile array, but only "${totalfile[0]}" and "${totalfile[-1]}" can print normally, while "${totalfile[1]}","${totalfile[2]}" and "${totalfile[3]}" does not print at all.

One more thing, when I use "${totalfile[-4]}","${totalfile[-3]}" and "${totalfile[-2]}", it works.

use -4,-3,-2, instead of 1,2,3

echo "the length of totalfile is: ${arraylength}"
echo "the 1st element: ${totalfile[0]}"
echo "the 2st element: ${totalfile[-4]}"
echo "the 3st element: ${totalfile[-3]}"
echo "the 4st element: ${totalfile[-2]}"
echo "the 5st element: ${totalfile[-1]}"

output:

the length of totalfile is: 5
the 1st element: /var/log/app/h323server.log.1584554401.gz
the 2st element: /var/log/app/h323server.log.1584945601.gz
the 3st element: /var/log/app/h323server.log.1584946801.gz
the 4st element: /var/log/app/h323server.log.1584948001.gz
the 5st element: /var/log/app/h323server.log.1584949201.gz

The os system is "Ubuntu 14.04.5 LTS".

I don't understand how it happens.And I'll be appreciated if anyone can explain it to me.

Answer 1

Storing state can be complicated in bash. Just parse the stream as it goes.

start_time='now -2 hour'
stop_time='now -1 hour'

# convert to seconds since epoch
start_time=$(date --date="$start_time" +%s)
stop_time=$(date --date="$stop_time" +%s)

# get list of files
( cd /var/log/app/ && find . -type f -name 'h323server.log.*.gz' ;) |
# extract the number
sed 's/\.\([0-9]*\).gz$/& \1/' |
# compare and print the filename
awk -v start_time="$start_time" -v stop_time="$stop_time" \
     'start_time < $2 && $2 < (stop_time + 20 * 60) { print $1 }' 
# I guess maybe also `(start_time - 20 * 60)` to fetch the previous one

Notes:

• Nice script!
• Use for ((i = 0; i < ${#array[@]}; ++i)) to iterate over array indexes. Or just for i in ${!array[@]}.
• I prefer arithmetic expansion, instead of if [[ ${filetime[${index}]} -ge ${timesys_s} ]] I would if (( ${filetime[${index}]} >= ${timesys_s} )).

Or for example get the file before and after the match:

find . -type f -name 'h323server.log.*.gz' |
# extract the number
sed 's/\.\([0-9]*\).gz$/& \1/' |
# sort on numbers
sort -n -k2 |
# important - the input is sorted
# compare and print the filename 
awk -v start_time="$start_time" -v stop_time="$stop_time" '
    # Because i dont want to write  stop_time > $2 && $2 > start_time everrywhere, I cache it in cond variable
    # clear cond variable
    { cond=0 }
    stop_time > $2 && $2 > start_time {
        cond_was_true=1; # remember that at least once the condition was met
        cond=1; # if the condition is met, set cond variable
    }
    # so, if the condition is met
    cond {
        # output the previous line before the match if any
        # if we did not output the previous line yet (oncelast)
        # and the previous line length is not empty
        if (!oncelast && length(last) != 0) {
            # remember that we ouputted the previous line and output it
            oncelast=1
            print last;
        }
        # output the current line
        print $1;
        # there is nothing interesting below
        next;
    }
    # remember the previous line
    # the !cond could be just removed, it want be executed because next above
    !cond { last=$1; }
    # print one more line after the condition is true
    # if the condition was true before
    # but is no longer true
    # then only once output the next line after the condition was met
    cond_was_true && !cond && !once { once=1; print $1; }
'

If you seem to want to print the content of resulting files, add | xargs -d$'\n' zcat on the end of the scripts.

After the sort -n -k2 the input is sorted using the timestamps. So we have a condition stop_time > $2 && $2 > start_time and I am interested in the one line before and one after the range the condition is met for the input.

Above I used cond variable to just not write stop_time > $2 && $2 > start_time over and over again. I guess I'll try to rewrite a simpler version, but untested:

awk -v start_time="$start_time" -v stop_time="$stop_time" '

    stop_time > $2 && $2 > start_time {
        # if the condition wasnt true, output the previous line
        if (!cond_was_true &&
               # and the previous_line is not empty
               length(previous_line) != 0) {
            print last;
        }
        # remember that the condition was true
        cond_was_true = 1;
        # output the current line
        print $1;
    }

    # remember the previous line
    { previous_line = $1; }

    # if the condition was true
    # but is no longer met
    # output the next line
    # but output it only once
    cond_was_true && 
             !(stop_time > $2 && $2 > start_time) &&
             !output_next_line_once { 
         output_next_line_once = 1;
         print $1;
    }
'

Answer 2

I'm thinking you should use readarray instead of a for loop to read the values: How to use 'readarray' in bash to read lines from a file into a 2D array

But if you just want to get the first and last lines, sed is likely a good option instead:

$ sed -n -e '1p' -e '$p' /etc/passwd
below cmd output started 2020 Mon Mar 23 08:19:32 AM PDT
root:x:0:0:root:/root:/bin/bash
apacheds:x:124:131::/var/lib/apacheds:/bin/bash

BTW, what do you require for a log file with only one line in it? Should it print the same line twice?