How to grep only the recently matched search string?

Question

I have a log file which gets updated every second. I am grep'ing some string and pulls that search result to a temp file. Then I am sending that temp file result to my email. I am running this script in cron. But when the cron triggers the script again and when the script catches the newly searched string, it gives me the previous/old result also in the temp file.

For example my log file looks like below but not exactly:

2018-02-15 14:36:47,344 INFO : Bread butter jam

2018-02-15 14:37:22,566 INFO : trees

2018-02-15 14:37:22,636 INFO : fruits

2018-02-15 14:37:22,636 INFO : veggies

2018-02-15 14:37:22,745 INFO : junkies

2018-02-15 14:37:23,648 INFO : Bread butter jam

2018-02-15 14:37:23,659 INFO : cakes

2018-02-15 14:37:23,734 INFO : cookies

2018-02-15 14:37:23,767 INFO : meat

2018-02-15 14:37:23,874 INFO : yogurt

I want Bread butter jam to be stored in a temp file each time when it comes in the log file.

How to pull only the newly searched result to the temp file?

Sorry for my bad English and I am new to bash.

Posting sample Input_file will help us to help you, please post it with CODE TAGS here. — RavinderSingh13, Feb 16 '18 at 10:14
Store line count of log file in some file and use `tail` to get lines from that line count + 1 — anubhava, Feb 16 '18 at 10:15
Remember when you ran the script the previous time and only search the log for entries after that point in time. See [extract data from log file in specified range of time](https://stackoverflow.com/questions/7575267/extract-data-from-log-file-in-specified-range-of-time) for details. — tripleee, Feb 16 '18 at 10:33
Stack Overflow is not a code writing service. Please show your code. Since Stack Overflow hides the Close reason from you: *Questions seeking debugging help ("why isn't this code working?") must include the desired behavior, a specific problem or error and the shortest code necessary to reproduce it in the question itself. Questions without a clear problem statement are not useful to other readers. See: [How to create a Minimal, Complete, and Verifiable example](http://stackoverflow.com/help/).* — jww, Feb 16 '18 at 10:51
Let `grep` number the lines for you, and let `tail` take only the last one. You can then immediately see if the new is different from the old because the line number will be different `grep -n "Bread" YourFile | tail -n 1` — Mark Setchell, Feb 16 '18 at 10:55
@MarkSetchell What if there are 3 occurences of "Bread" between the 2 logs ? You'd only get the last one — Aserre, Feb 16 '18 at 11:01
@Aserre I take your point - though I am not actually sure what the OP wants, so I just made a comment instead of an answer :-) — Mark Setchell, Feb 16 '18 at 11:51

score 2 · Answer 1 · answered Feb 16 '18 at 11:12

As mentionned in the comments, you should compare the current log size with the length of the previous one you would have stored in a file.

Something like that should do the trick :

#!/bin/bash
CURRENT_LINECOUNT=$(cat /path/to/LogFile | wc - l)
#redirection is here in case the old_count file doesn't exist
OLD_LINECOUNT=$(cat /path/to/old_count 2>/dev/null)

tail -n $((CURRENT_LINECOUNT - ${OLD_LINECOUNT:-0})) /path/to/LogFile | grep "Bread butter jam" > /path/to/temp/file
echo $CURRENT_LINECOUNT > /path/to/old_count

#here, your logic to send the temp file

score 1 · Answer 2 · answered Feb 16 '18 at 23:19

When the logfile is small enough for grepping strings each minute, you can use the fact that new lines will have a different timestamp. Something like

mytmp=/tmp/breakfast.tmp
mylasttmp=/tmp/breakfast.lasttmp
myattachment=/tmp/breakfast.now
test -f ${mytmp} && { echo "Last cron processing still running"; exit 1;}
touch ${mytmp}

grep -E "Bread butter jam" logfile > ${mytmp}

comm -3 ${mytmp} ${mylasttmp} > ${myattachment}
# process ${myattachment} and when that is finished...
mv ${myattachment} ${mylasttmp}
rm ${mytmp}

How to grep only the recently matched search string?

2 Answers2