1

I have many log files and I have a few log4j configurations (different ConversionPattern formats for file appenders).

I would like to write a script/test which is able to group log files by a ConversionPattern which was used to write a file. The bit I'm missing is how to implement something like:

boolean logToPatternMatcher.matches(String conversionPattern, String logFileEntry)

Is it possible with Log4j API?

The above method should return true for a given:

String conversionPattern = "%d{ISO8601} %-5p [%-16.16t][%c] %m%n"

String logFileEntry = "2015-02-12 00:02:38,023 WARN [pool-58-thread-1][some.package.name.ConvertingPublisher] Document [type: app.MessageProcessed, id: 1063_1_20150128072222800] DUPLICATED."

topr
  • 4,482
  • 3
  • 28
  • 35

1 Answers1

0

GROK is perfect for your use case. I have been using it with logstash recently but there is a java-grok project on github you could try. I found this link from another stackoverflow answer.

GROK is essentially a regexp engine. It has a bunch of pre-defined regexp's that will match most common logging entries like timestamps, logging level etc. If that doesn't work out for you have you thought about using straight regexp?

Community
  • 1
  • 1
Gwyn Jensen
  • 55
  • 1
  • 1
  • 5
  • Thanks, I'll have a look at GROK. Problem with regexps is that I don't have regexps. Thus I would need to translate log4j patterns (which I do have) into regexps. This of course is a possible solution but seems like going around. If log4j is able to produce specific output text out of `LogEvent` with a given `ConversionPattern` its API should be able to do the opposite (kind of deserialize). Conversion patterns are meant for further logs analysis not only writing them. There is plenty of such software with which you can configure log parsers just by providing log4j pattern. – topr Feb 12 '15 at 18:07
  • 1
    @topr: You may want to look at [LogMX](http://www.logmx.com), it directly parses logs from a Log4j pattern, and auto-detects the suitable Parser for your file. You simply have to create a Parser (from your Pattern) for each log file type, then LogMX will tell you which Parser it has used to open a file. Yet, it is not something you can embed in your code. – xav Feb 14 '15 at 12:34
  • @xav, thanks mate. I knew about LogMX and I'm using it. Actually I used it for this job as well, but in really manual kind of way. In parser configuration I was pasting log samples from each file and the different patterns to see whether it parses fine. It's not the way to go with bigger volumes of log files and patterns though. Too tedious to do it by hand like that. – topr Feb 16 '15 at 17:19