50

I need to read through some gigantic log files on a Linux system. There's a lot of clutter in the logs. At the moment I'm doing something like this:

cat logfile.txt | grep -v "IgnoreThis\|IgnoreThat" | less

But it's cumbersome -- every time I want to add another filter, I need to quit less and edit the command line. Some of the filters are relatively complicated and may be multi-line.

I'd like some way to apply filters as I am reading through the log, and a way to save these filters somewhere.

Is there a tool that can do this for me? I can't install new software so hopefully it's something that would already be installed -- e.g., less, vi, something in a Python or Perl lib, etc.

Changing the code that generates the log to generate less is not an option.

gareth_bowles
  • 20,760
  • 5
  • 52
  • 82
Dan
  • 5,929
  • 6
  • 42
  • 52
  • 1
    I don't have a magic wand for you, but this might be beter on serverfault.com... – Peter Loron Feb 26 '10 at 01:01
  • 1
    @Peter -- good suggestion: http://serverfault.com/questions/117013/how-can-i-view-log-files-in-linux-and-apply-custom-filers-while-viewing – Dan Feb 26 '10 at 01:34
  • 1
    Note that it's considered bad style to start a pipeline with `cat FILE | ...`. Better would be `grep args < FILE | ...` or just `grep args FILE | ...` – R Samuel Klatchko Feb 26 '10 at 01:48
  • @RSK: Normally I would do something like that, but in reality I have a bunch of grep's piped together and it seemed silly to write: `grep -v "OneFilter" < FILE | grep -v "AnotherUglyLongFilter" | grep -v "etc." | less` ... I guess just because it buries the filename a bit more. – Dan Feb 26 '10 at 01:54
  • 1
    Actually the style of using `cat FILE | grep .. | awk ... | sort` is more clear. The problem is not style, it is performance, as this way one more process is created and more inter-process communication happens. This is usually no problem for interactive commands, but in scripts the more ugly form should be used. – Thraidh Mar 04 '13 at 13:28

5 Answers5

115

Use &pattern command within less.

From the man page for less

&pattern

          Display  only  lines which match the pattern; lines which do not
          match the pattern are not displayed.  If pattern  is  empty  (if
          you  type  &  immediately  followed  by ENTER), any filtering is
          turned off, and all lines are displayed.  While filtering is  in
          effect,  an  ampersand  is  displayed  at  the  beginning of the
          prompt, as a reminder that some lines in the file may be hidden.

          Certain characters are special as in the / command:

          ^N or !
                 Display only lines which do NOT match the pattern.

          ^R     Don't interpret regular expression  metacharacters;  that
                 is, do a simple textual comparison.
Rob Kielty
  • 7,958
  • 8
  • 39
  • 51
ALF
  • 1,151
  • 1
  • 7
  • 3
  • 1
    Neat feature. Not helpful for what I was doing 2 years ago but I'll definitely use that one in the future! – Dan Jul 04 '12 at 14:31
  • 3
    This is exactly what I was looking for when I found this question. And I think it's exactly what the question owner asked for. This should be the accepted answer! Anyway, many thanks, you saved me a lot of time! – Marcin Koziński Jul 25 '12 at 08:31
  • @krookedking upgrade to a newer version of less. I had the same problem and upgraded with `brew install less`, got version 458 which support it. – ordahan Jan 31 '16 at 12:09
4

Try the multitail tool - as well as letting you view multile logs at once, I'm pretty sure it lets you apply regex filters interactively.

Ty.Sp
  • 7
  • 3
gareth_bowles
  • 20,760
  • 5
  • 52
  • 82
  • 1
    That looks great... unfortunately not installed. Maybe I'll have to beg the sysadmin. – Dan Feb 26 '10 at 01:38
4

Based on ghostdog74's answer and the less manpage, I came up with this:

~/.bashrc:

export LESSOPEN='|~/less-filter.sh %s'
export LESS=-R  # to allow ANSI colors

~/less-filter.sh:

#!/bin/sh
case "$1" in
*logfile*.log*) ~/less-filter.sed < $1
  ;;
esac

~/less-filter.sed:

/deleteLinesLikeThis/d  # to filter out lines
s/this/that/  # to change text on lines (useful to colorize using ANSI escapes)

Then:

  • less logfileFooBar.log.1 -- applies the filter applies automatically.
  • cat logfileFooBar.log.1 | less -- to see the log without filtering

This is adequate for now but I would still like to be able to edit the filters on the fly.

Community
  • 1
  • 1
Dan
  • 5,929
  • 6
  • 42
  • 52
0

see the man page of less. there are some options you can use to search for words for example. It has line editing mode as well.

ghostdog74
  • 327,991
  • 56
  • 259
  • 343
  • I need to filter, not search. The INPUT PREPROCESSOR may be helpful, although it's not as dynamic as I wanted. – Dan Feb 26 '10 at 01:47
0

There's an application by Casstor Software Solutions called LogFilter (www.casstor.com) that can edit Windows/Mac/Linux text files and can easily perform file filtering. It supports multiple filters as well as regular expressions. I think it might be what you're looking for.

Ben
  • 1