Shell Scripting (Extracting the logs for last 24 hours)

Question

I have a huge logfile1 (Linux) with below date format & I want to extract the past 24 hrs of data & write into different file with a shell script. Please help me to achieve the task?

----------
03/03/2016 05:40:42 AM QWTRAB1 AMQ7315: Failed to put message to accounting queue. Reason(2053
----------
03/03/2016 05:40:42 AM QWTRAB1 AMQ7315: Failed to put message to accounting queue. Reason(2053
----------
03/03/2016 05:40:46 AM QWTRAB1 AMQ7315: Failed to put message to accounting queue. Reason(2053
----------
03/03/2016 05:40:46 AM QWTRAB1 AMQ7315: Failed to put message to accounting queue. Reason(2053

score 1 · Answer 1 · edited May 23 '17 at 11:50

1

you can do something similar to Split access.log file by dates using command line tools:

split.awk:

{
    split($1,array,"[:/]");
    year = array[3]
    month = array[2]
    day = array[1]

    print > FILENAME"-"year"_"month"_"day".txt"
}

command:

awk -f split.awk your_log_file.log

edited May 23 '17 at 11:50

Community

1
1

answered Mar 03 '16 at 13:22

MaxU - stand with Ukraine

205,989
36
386
419

./test.sh + awk $'BEGIN {\n split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec ", months, " ")\n for (a = 1; a <= 12; a++)\n m[months[a]] = a\n}\n{\n split($4,array,"[:/]");\n year = array[3]\n month = sprintf("%02d", m[array[2]])\n\n print > /apphome/mqm/logs/NewFile"-"year"_"month".txt"\n}' /apphome/mqm/logs/MQLOGS.OUT.0303160800.txt awk: cmd. line:10: (FILENAME=/apphome/mqm/logs/MQLOGS.OUT.0303160800.txt FNR=1) fatal: division by zero attempted – Krishna Mar 03 '16 at 13:40
@Krishna, i've updated my answer. If you really have '------' lines in your log file you would have to filter them out first... – MaxU - stand with Ukraine Mar 03 '16 at 14:16

Rocoder · Answer 2 · 2016-03-04T08:39:47.450

0

Try awk command providing the date range

awk '$0 >= "03/03/2016 05:40" && $0 <= "03/02/2016 05:40"'

UPDATE:

#!/bin/bash
position=0
length=10
PREVDATE=`date -d "-1 days" +"%d-%m-%Y"`
while IFS='' read -r line || [[ -n "$line" ]]; do
        LOGDATE=${line:position:length}
        if [ "$PREVDATE" = "$LOGDATE" ]; then
                echo $line >> $PREVDATE.log
        fi;     
done < "$1"

If you want to get the log for previous day. You can run this script in crontab everyday once. Save the given script in a file called logparser.sh. Provide the main log file as argument. You can find the parsed data in new log file name with previous date.

try $./logparser.sh logfilename.log

edited Mar 04 '16 at 08:39

answered Mar 03 '16 at 13:26

Rocoder

1,083
2
15
26

I want to do this on daily basis with a cron job – Krishna Mar 03 '16 at 13:42
@Krishna I have updated the answer. You can run this script as crontab !! – Rocoder Mar 04 '16 at 08:41
Looks like it's not giving the output in write format – Krishna Mar 07 '16 at 15:55
+ IFS=' + LOGDATE=03/07/2016 + [ 06-03-2016 '= 03/07/2016 ] + read -r line + IFS=' + LOGDATE=03/07/2016 + [ 06-03-2016 '= 03/07/2016 ] + read -r line + IFS=' + LOGDATE=03/07/2016 + [ 06-03-2016 '= 03/07/2016 ] + read -r line + IFS=' + [[ -n ' ]] – Krishna Mar 07 '16 at 15:56
Looks like it's not giving the output in right format , when I try to execute the logpraser.sh with the logfilename.log as a argument – Krishna Mar 07 '16 at 16:02
@Krishna if you run everyday. It will parse the yesterday's log data into file. You can modify the DATE part as per you requirement. !! I did it like that bcoz if you run it once a day it will parse ur all the log of yesterday's date. – Rocoder Mar 12 '16 at 08:38

Shell Scripting (Extracting the logs for last 24 hours)

2 Answers2