Delete lines prior to a specific date in a log file

Question

I have a ffmpeg process log file in which log data get appended to it daily. I want to do a automatic maintainence of this log by going through the log and deleting lines belonging to a certain date. I tried something but it didn't help.

How should i do it? Please help. Thanks.

Example. To delete all lines prior to Jun 01.

#!/bin/bash

var=`date -d "now-30 day" | awk '{print $3}'`
path="/home/pi/scripts/"

## Removing Entries before 30 Days ####
for logs in $path/process.log; do

awk -v rot=$var '{ if ($NF > rot ) {print $0}}' $logs > $path/temp.log
mv $path/temp.log $logs

done

Example of process.log file which contains data prior to june i want to delete all the previous data.

Oct 28 06:48:15 #####################################Start Time########################################

Stopping Transaction Video Recording
ffmpeg version 4.2.1-static https://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libfribidi --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libxml2 --enable-libxvid --enable-libzimg
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
/dev/v4l/by-id/usb-WN-720P-HD_WN-720P-HD_WN-720P-HD-video-index0: Device or resource busy

Oct 28 06:48:17 #####################################End Time########################################


Oct 29 07:38:13 #####################################Start Time########################################

Stopping Transaction Video Recording
ffmpeg version 4.2.1-static https://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libfribidi --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libxml2 --enable-libxvid --enable-libzimg
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
/dev/v4l/by-id/usb-WN-720P-HD_WN-720P-HD_WN-720P-HD-video-index0: Device or resource busy

Oct 29 07:38:13 #####################################End Time########################################"

This is the data i want to preserve which is of current month's data:

Jun 02 09:49:08 #####################################Start Time########################################

Stopping Transaction Video Recording
ffmpeg version 4.2.1-static https://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libfribidi --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libxml2 --enable-libxvid --enable-libzimg
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[video4linux2,v4l2 @ 0x36e75e0] The driver changed the time per frame from 1/5 to 1/30
[mjpeg @ 0x36e8530] EOI missing, emulating
Input #0, video4linux2,v4l2, from '/dev/v4l/by-id/usb-WN-720P-HD_WN-720P-HD_WN-720P-HD-video-index0':
  Duration: N/A, start: 13731.683520, bitrate: N/A
    Stream #0:0: Video: mjpeg (Baseline), yuvj422p(pc, bt470bg/unknown/unknown), 1280x720, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> mpeg4 (native))
Press [q] to stop, [?] for help

@SubhamSahu : At least with my version of `date`, your way to set `var` would store the **year** into `var`. — user1934428, Jun 18 '21 at 07:17
You have a structural problem: your log files mention dates without including the year. How will you know some entry is from a previous year? A workaroud is to delete also the entries from the future assuming these were entries from a past year, but still then, you will miss some entries you have to delete from the past. — Pierre François, Jun 18 '21 at 13:40

Darkman · Answer 1 · 2021-06-18T13:04:49.787

0

I'd use sed.

sed -i.bak '/^Jun 01/,/^Jun 01.*$\n/!d' log.txt

This will create a backup of log.txt named log.txt.bak. But still, I'd suggest you to make a backup first with a different name.
This assumes that Jun 01 exists in the log.txt.

EDITED

#!/bin/bash

target_file='log.txt'
target_date='Jun 01'

line=$(awk -v date="$target_date" '
    date==$1" "$2 {
        if(n++==0) { line=FNR
        } else { n=0 }
    } END {
        print line-1
    }' "$target_file")

if [[ "$line" > 1 ]]; then
    sed -i.bak "1,${line}d" "$target_file"
fi

Similar as the above command, It will create a backup file named e.g. log.txt.bak. Change the target_file and target_date according to your needs. This script will delete all lines prior date even though there are multiple duplicates.

edited Jun 18 '21 at 13:04

answered Jun 18 '21 at 09:04

Darkman

2,941
2
9
14

The use of `sed` here is legitimate, but in that case, you need to find a system for selecting not only the sections corresponding to `Jun 01`, as you do above, but also to select the sections corresponding to older dates, as far as I understand the question of the PO. – Pierre François Jun 18 '21 at 10:55
As Pierre François said; unless you have unique dates, the above command is not that helpful. – Darkman Jun 18 '21 at 12:06

score 0 · Answer 2 · answered Jun 18 '21 at 14:41

We miss elements for coding waterproof. We need to know a.o. if it possible to find entries from past years in the log files, because the log files don't mention the year of the entries, only the month and the day.

We suppose that a date apparently in the future corresponds to entries of former years. If you run the script in the beginning of January, you will delete entries of December which are younger than 30 days.

This implies the problem can't be solved in a completely satisfactory way without more knowledge. Sometimes the name of the logfiles can contain some useful information about the year that we ignore here.

Anyway, the solution below may work, but it is not very optimal:

#!/bin/bash

#time stamp older entries to keep
ts_now=$(date +"%s" -d now)
ts_oldest_to_keep=$(date +"%s" -d "now - 30 days")
#adapt path to point to the directory where log files are living
path="."

for ts in $(grep "Start Time" "$path/"*".log" | awk '{print $1, $2}' | date +"%s" -f - | sort -u)
do 
  # get the timestamp corresponding to all the entries to be removed
  if [ $ts -lt $ts_oldest_to_keep ] || [ $ts -gt $ts_now ]
  then 
    # convert the timestamp into the string "%b %d" i.e. the abbreviated month 
    # name in English followed by space and zero padded day of month.
    # LC_ALL=C to make sure you get an English name of month
    ym=$(LC_ALL=C date +"%b %d" -d "@$ts") 
    # delete the block of lines beginning with "$ym" and ending with 
    # the same string
    sed -i "/^$ym.*Start Time/,/^$ym.*End Time/d" "$path/"*".log"
  fi
done

You first grab all the possible dates from the log files. You convert the date into a time stamp for making a numerical comparison of dates possible, and you use sed to delete block of lines without the need of a temporary file.

Looking for a solution in AWK in one pass shoud also be possible, but AWK misses a standard function to convert easily dates as Jun 01.

Anyway, backup your log files before messing around.

Delete lines prior to a specific date in a log file

2 Answers2