59

I am looking for some kind of analyzer tool for log files generated by log4j files. I am looking something more advanced than grep? What are you using for log file analysis?

I am looking for following kinds of features:

  • The tool should tell me how many time a given log statement or a stack trace has occurred, preferably with support for some kinds of patterns (eg. number of log statements matching 'User [a-z]* logged in').
  • Breakdowns by log level (how many INFO, DEBUG lines) and by class that initiated the log message would be nice.
  • Breakdown by date (how many log statements in given time period)
  • What log lines occur commonly together?
  • Support for several files since I am using log rolling
  • Hot spot analysis: find if there is a some time period when there is unusually high number of log statements
  • Either command-line or GUI are fine
  • Open Source is preferred but I am also interested in commercial offerings

My log4j configuration uses org.apache.log4j.PatternLayout with pattern %d %p %c - %m%n but that could be adapted for analyzer tool.

Juha Syrjälä
  • 33,425
  • 31
  • 131
  • 183

9 Answers9

18

(disclaimer: I'm one of the developers contributing to Chainsaw V2)

Chainsaw V2 can provide some of the functionality you're looking for through its support for custom expressions and the ability to use those expressions to colorize, search and filter events.

You -can- load multiple log files into Chainsaw (by default, all events for a log file are placed on a logfile-specific tab). You can also define a 'custom expression logpanel' which will aggregate events from all tabs into a new tab matching an expression you provided - similar to a database 'view', you could use the expression 'LEVEL >= WARN' to collect all warnings, error & fatal messages from any log file into a single view.

Some example expressions which could be used to colorize, search or filter events:

  • msg like 'User [a-z]* logged in'
  • msg ~= login || msg ~= logout
  • level > INFO
  • exception exists
  • timestamp <= '2010/04/06 15:05:35'

The only way to get 'counts' currently is to define an expression in the 'refine focus' field (the count of events matching the expression will show in the status bar).

One of the useful features added to the upcoming release is a clickable bar to the right of the table (similar to Eclipse or Idea's bar showing syntax error indications) which will display color rule and search expression matches for the entire log file.

When the next version of Chainsaw V2 comes out, I hope you give it a spin - it's Open Source, free, and we're always interested in suggestions & feedback.

A. Tapper
  • 1,261
  • 8
  • 17
Scott
  • 1,728
  • 11
  • 11
  • 1
    Does chainsaw work without XMLLayout? With normal PatternLayout? – Juha Syrjälä Apr 08 '10 at 09:10
  • Yes, using LogFilePatternReceiver. You can find javadoc from Chainsaw's help menu, and an example config file on the Welcome tab (the view example receiver configuration button). The 'logFormat' for your pattern is: TIMESTAMP LEVEL LOGGER - MESSAGE You can also use VFSLogFilePatternReceiver, which gives you access to the Jakarta Commons-VFS filesystems (tail files over ssh, etc). I suggest using the VFS receiver, (tailing is more reliable) - just add the necessary jars (vfs, jsch, commons logging, etc) to the $user/.chainsaw/plugins folder or to your classpath). – Scott Apr 08 '10 at 15:58
  • Last night I added the search match count to the status bar, so you may find that useful (you can now get counts via filtering and searching, in the status bar). – Scott Apr 08 '10 at 16:01
  • Hi @Scott I am trying to get Chainsaw to work with my server logs, our logs typically go through a custom file appender. Now to start easy, I tried to just pointed chainsaw at one of the files using the LogFilePatternReceiver, and after hardcoding the config to some values to point to one of my files, I still can't see the logs in chainsaw. The tutorial works fine, I'm sure I'm missing something. Any ideas? – Charbel Nov 13 '14 at 11:19
  • Probably easiest to help you on the log4j users mailing list, but if you're using an sftp:// path, make sure you can ssh to the server with the credentials you're using in the Chainsaw config. If Chainsaw can connect to the server and find the file, even if it can't parse the file due to the log format, you'll get the log messages in the table, just not parsed by logger name etc (the entire line will be in the 'message' field with an 'unknown' logger). – Scott Nov 13 '14 at 16:29
  • 1
    It does nt work. Error - Failed to validate certificate. – Jay Mar 04 '16 at 16:49
  • Come on this tool was last updated 10 years ago, and also has an out-of-date certificate. This answer is no longer relevant :-) – matanster Jun 15 '16 at 14:53
  • 3
    Last released a long time ago, but the developer snapshot has tons of new features and is much better: https://people.apache.org/~sdeboy/ – Scott Jun 15 '16 at 18:08
9

I'd suggest Splunk. It provides fast, Google-like searching across lots (terabytes) of logs, is easy to filter (e.g. by log level or date), makes it easy to correlate into transactions of multiple related log events, etc.

There's a downloadable version that's free as long as you're indexing less than 500MB of logs per day.

Justin Grant
  • 44,807
  • 15
  • 124
  • 208
  • That seem promising. Any idea about pricing? – Juha Syrjälä Apr 07 '10 at 06:38
  • 4
    There's a free version that handles indexing 500MB per day of logs. If you're logging more than 500MB/day, Splunk isn't cheap (thousands of dollars) but most users logging that much data every day are typically running a large, expensive site anyways, so cost may be less of an issue. – Justin Grant Apr 07 '10 at 15:41
2

Might come a bit late, but LogMX does all this stuff, and is highly active for many years now. It is not open-source but it is powerful even if it doesn't seem to!

xav
  • 5,452
  • 7
  • 48
  • 57
2

Take a look at Apache Chainsaw http://logging.apache.org/chainsaw/index.html for your needs

JoseK
  • 31,141
  • 14
  • 104
  • 131
2

You can try LogSaw, it's an open source software based on Eclipse and which is active right now...

Anthony O.
  • 22,041
  • 18
  • 107
  • 163
1

I have created a custom tool for that: https://plus.google.com/u/0/102275357970232913798/posts/Fsu6qftH2ja

Alfa is a GUI tool for analizing log files. Usually you are forced to search for data in them using editors. You open a log, press Ctrl-F and the "Next" button again and again, then reload the file as it was modified, and repeat the search. Alfa maps a log file to a database allowing you to use standard SQL queries to get data without any superfluous actions.

Dzmitry Krakadzeyau
  • 11
  • 1
1

Mind Tree Insight is also a useful Open Source Log Analysis tool

http://sourceforge.net/projects/mindtreeinsight

Abhi
  • 6,471
  • 6
  • 40
  • 57
  • 1
    Be careful when downloading. Sophos just reported it to be a virus/malware. Not sure if it's correct, but I removed it immediately, just to be safe... – Bhaskar Feb 17 '17 at 07:31
0

you can also try an Online log file analysis-

http://www.sharontools.com/tools/LogAnalysis/Main.php

Smith

Smith
  • 1
  • 1
    I found this one to be very shaky and had a lot of errors. The Help link didn't work. Looks like a neat idea, but the execution is at this point poor. – luiscolorado Aug 01 '12 at 16:21
0

glogg is a simple but powerful tool. It allows to color lines by filter expressions and has breakpoint-style markers. A separate panel shows search results and/or markers.

glogg screenshot

Falko Menge
  • 788
  • 7
  • 16
  • 1
    Glogg seems to be abandoned, no updates since 2017. There is an active fork -- [klogg](https://github.com/variar/klogg) – fav May 07 '21 at 22:48