Parse Apache2 Error logs with Grok for Logstash

Question

Im trying to parse my apache2 error log and im having a bit of trouble.. It doesnt seem to be matching the filter. Im pretty sure the timestamp piece is wrong, but im not sure, and i cant really find any documentation to figure it out. Also, is there a way to get what is in fields.errmsg to me @message?

Log

[Wed Jun 26 22:13:22 2013] [error] [client 10.10.10.100] PHP Fatal error:  Uncaught exception '\Foo\Bar'

Shipper Config

input {
        file {
                'path' => '/var/log/apache2/*-error.log'
                'type' => 'apache-error'
        }

}

filter {
        grok {
                type => "apache-error"
                pattern => "\[%{HTTPDATE:timestamp}\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}"
        }
}


output {
        stdout {}
        redis {
                'data_type' => 'list'
                'host' => 'logstash.server.net'
                'key' => 'logstash'
        }
}

score 31 · Accepted Answer · answered Feb 20 '14 at 04:09

Ahoy!

I know I'm a little late to the party, but here it goes!

I created a /etc/logstash/patterns.d/ directory on the system and thew a file named apache-error in it containing:

APACHE_ERROR_TIME %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
APACHE_ERROR_LOG \[%{APACHE_ERROR_TIME:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:errormsg}

/etc/logstash/patterns.d/ will be referenced in the logstash configuration as follows:

grok {
  patterns_dir => [ "/etc/logstash/patterns.d" ]
  match => [ "message", "%{APACHE_ERROR_LOG}" ]
}

You can test it out at Grok Debugger, like Adam mentioned in his comment. Seems to work just fine with the sample log entry you send. Has been working pretty solidly me with one of my clients.

The pattern above puts the final message in errormsg field. So I just delete the message field.

This is what I am currently using in my logstash configuration:

filter {
  if [type] == "apache_error_log" {

    grok {
      patterns_dir => [ "/etc/logstash/patterns.d" ]
      match => [ "message", "%{APACHE_ERROR_LOG}" ]
    }

    if !("_grokparsefailure" in [tags]) {

      mutate {
        remove_field => [ "message" ]
        add_field =>  ["timestamp_submitted", "%{@timestamp}"]
      }

      date {
        # Try to pull the timestamp from the 'timestamp' field (parsed above with
        # grok). The apache time format looks like: "18/Aug/2011:05:44:34 -0700"
        #                        Sat Feb 08 06:31:09 2014
        match => [ "timestamp", "EEE MMM dd HH:mm:ss yyyy" ]
        remove_field => [ "timestamp" ]
      }

      geoip {
        source => "clientip"
      }
    }
  }
}

Note that I am using a type of apache_error_log instead of apache-error.

Give it s shot. I would love to know if this works for you and others!

Cheers!

Thanks! One quick note: I'm using the GELF output plugin to send logs to Graylog2. It requires the message field not be missing, so I had to adjust the pattern accordingly. — Zach Lipton, Jul 03 '14 at 19:24
I'm guessing the log error log format changed since this post, but in case it'll help someone I updated the APACHE_ERROR_LOG pattern to look like this instead: `APACHE_ERROR_LOG \[%{APACHE_ERROR_TIME:timestamp}\] \[:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}\] (?:\[client %{IPORHOST:clientip}:%{POSINT:port}\] ){0,1}%{GREEDYDATA:errormsg}` — Marco, Aug 13 '15 at 12:13
Does not seem to be working for me.. `[Sun Oct 11 23:38:09.105800 2015] [mpm_event:notice] [pid 21264:tid 140035264497536] AH00489: Apache/2.4.7 (Ubuntu) configured -- resuming normal operations` — Mandeep Singh, Oct 20 '15 at 19:01
This one worked for me `APACHE_ERROR_LOG \[%{APACHE_ERROR_TIME:timestamp}\] \[%{WORD:source}:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}:tid %{NUMBER:tid}\] (?:\[client %{IPORHOST:clientip}:%{POSINT:port}\] ){0,1}%{GREEDYDATA:errormsg}` — Mandeep Singh, Oct 20 '15 at 19:20
in case anyone else also runs into this, I needed to add an `ssl` in there: `APACHE_ERROR_TIME %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR} APACHE_ERROR_LOG \[%{APACHE_ERROR_TIME:timestamp}\] \[ssl:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}\] (?:\[client %{IPORHOST:clientip}:%{POSINT:port}\] ){0,1}%{GREEDYDATA:errormsg}` — gsaslis, Oct 09 '16 at 18:56

score 12 · Answer 2 · answered Dec 19 '16 at 18:17

12

Our error logs look a little different:

[Tue Dec 08 12:30:35.997887 2015] [ssl:info] [pid 1178:tid 1072] (70014)End of file found: [client 10.129.24.77:49987] AH01991: SSL input filter read failed.

But I found a predefined pattern that works perfectly:

HTTPD24_ERRORLOG

See this in the Logstash source

answered Dec 19 '16 at 18:17

Robbie Aldrich

135
1
7

1

this is better approach – Asraful Mar 27 '17 at 09:13
worth noting that `HTTPD_ERRORLOG` will work for both 2.0 and 2.4 formats, so if you're shipping logstash with ansible to a heterogeneous environment (like me) then you should probably use the catch-all version that works for both. – Chris Browne Sep 11 '19 at 08:14
@Robbie Can you please show your filter config, how you have used it? – Tech Geek Oct 11 '20 at 20:35

score 4 · Answer 3 · answered Mar 04 '17 at 12:53

4

Didn't work for me:

EEE MMM dd HH:mm:ss yyyy

Worked for me (Apache 2.4):

EEE MMM dd HH:mm:ss.SSSSSS yyyy

answered Mar 04 '17 at 12:53

Detzler

162
2
7

score 2 · Answer 4 · answered Jun 28 '13 at 10:40

2

Logstash has a built-in apache log parser.

Here is an example...

grok {
   type    => 'company'
   pattern => ["%{COMBINEDAPACHELOG}"]
   add_tag => "apache"
}

As a reference, you can check Logstash's docs

answered Jun 28 '13 at 10:40

Adam

1,962
2
17
30

5

Thanks for pointing that out - didn't realize that. You should use the GrokDebugger to check your Grok patterns before putting them in your config file. It's pretty self-explanatory. http://grokdebug.herokuapp.com/ – Adam Jun 28 '13 at 15:28
nice. That helps. Any idea how to get it so `@message` is the same as `@fields.message` tho? – Ascherer Jun 28 '13 at 17:22
I'm not sure how that works. I know you can update fields' values by using something like `replace => [ "@message", "%{message_remainder}" ]`, but I don't know how to prepend the `@fields` to a new `message` variable. Out of interest, what would you use that for? I can't think of a use case off the top of my head. – Adam Jun 28 '13 at 20:03
i dont want the whole log line to be `@message`, i just want the actual error – Ascherer Jun 28 '13 at 20:11
The message itself will always be stored in @message, unless you change this using the `replace` function. I would recommend continuing on the path you are going (from your snippet in the question). It is a good idea to create a new variable `errmsg` and then to access the error message just look at `@fields.errmsg` – Adam Jun 28 '13 at 20:58

Parse Apache2 Error logs with Grok for Logstash

4 Answers4

Linked