Log4j logging directly to elasticsearch server

Question

I'm a bit confused on how can I put my log entries directly to elasticsearch (not logstash). So far I found a few appenders (log4j.appender.SocketAppender, log4j.appender.server etc.) that allow to send logs to remote host and also ConversionPattern possibility that seems to allow us to convert logs to "elastic-friendly" format, but this approach looks freaky... or do I mistake? Is this the one way to send logs to elastic?

So far I have a such config:

log4j.rootLogger=DEBUG, server
log4j.appender.server=org.apache.log4j.net.SocketAppender
log4j.appender.server.Port=9200
log4j.appender.server.RemoteHost=localhost
log4j.appender.server.ReconnectionDelay=10000
log4j.appender.server.layout.ConversionPattern={"debug_level":"%p","debug_timestamp":"%d{ISO8601}","debug_thread":"%t","debug_file":"%F", "debug_line":"%L","debug_message":"%m"}%n

But I get an error:

log4j:WARN Detected problem with connection: java.net.SocketException: Broken pipe (Write failed)

I can't find any useful example so I can't understand what do I do wrong and how to fix it. Thanks.

That's not possible. You need Logstash for that. If you want to use the deprecated log4j plugin (https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html) it's ok, but we recommend writing log4j logs to files and use Filebeat to pick them up and send them to Elasticsearch. — Andrei Stefan, May 26 '17 at 10:30
see this question: https://stackoverflow.com/questions/32302421/logging-from-java-app-to-elk-without-need-for-parsing-logs — diginoise, May 26 '17 at 10:44
@diginoise `If you point logstash to this file`.. .so this means I'll write logs to a file and that's not what I need — Frankie Drake, May 26 '17 at 11:08

score 5 · Answer 1 · answered Oct 15 '17 at 00:09

5

I've written this appender here Log4J2 Elastic REST Appender if you want to use it. It has the ability to buffer log events based on time and/or number of events before sending it to Elastic (using the _bulk API so that it sends it all in one go). It has been published to Maven Central so it's pretty straight forward.

answered Oct 15 '17 at 00:09

Marcelo Grossi

103
2
7

But it's not released yet? – Thomas Decaux Jun 05 '18 at 07:06
I can't find it on Maven Central. Could you please share a link? – Kira Resari Sep 27 '22 at 09:37

score 4 · Answer 2 · answered Jun 19 '18 at 17:19

4

If you'd like to check out something new, my Log4j2 Elasticsearch Appenders will give you async logging in batches with failover.

answered Jun 19 '18 at 17:19

rfoltyns

389
3
10

Thank you, `graylog` is absolutely satisfying for me – Frankie Drake Jun 20 '18 at 06:36
But that one has a vulnerability (see https://mvnrepository.com/artifact/org.appenders.log4j/log4j2-elasticsearch-core/1.5.5 )... Is that going to be fixed sometime? – Kira Resari Sep 27 '22 at 09:42
@KiraResari, (easily replaceable) dependency has a vulnerability, not the library itself. Fixed in 1.5.6. – rfoltyns Oct 05 '22 at 07:57

score 1 · Accepted Answer · answered Nov 02 '17 at 20:43

I found solution that fits my requirements most. It's a graylog . Since it's build based on elasticsearch the usage is familiar so I was able to switch to it immediately.

To use it I added this dependency along with basic log4j2 dependencies:

<dependency>
    <groupId>org.graylog2.log4j2</groupId>
    <artifactId>log4j2-gelf</artifactId>
    <version>1.3.2</version>
</dependency>

and use log4j2.json configuration:

{
  "configuration": {
    "status": "info",
    "name": "LOGGER",
    "packages": "org.graylog2.log4j2",
    "appenders": {
      "GELF": {
        "name": "GELF",
        "server": "log.myapp.com",
        "port": "12201",
        "hostName": "my-awsome-app",
        "JSONLayout": {
          "compact": "false",
          "locationInfo": "true",
          "complete": "true",
          "eventEol": "true",
          "properties": "true",
          "propertiesAsList": "true"
        },
        "ThresholdFilter": {
          "level": "info"
        }
      }
    },
    "loggers": {
      "logger": [
        {
          "name": "io.netty",
          "level": "info",
          "additivity": "false",
          "AppenderRef": {
            "ref": "GELF"
          }
        }        
      ],
      "root": {
        "level": "info",
        "AppenderRef": [
          {
            "ref": "GELF"
          }
        ]
      }
    }
  }
}

score 0 · Answer 4 · answered Dec 14 '20 at 23:16

0

You can send your logs by HTTP, with HttpAppender, see Log4j Appenders

answered Dec 14 '20 at 23:16

Кирилл Тимофеев

45
9

I'm sure, person who investigates oportunity to store his app logs with elasticsearch and integrate log4j with it, is aware of such thing as http logging. Inappropriate as an answer to the question. The worst approach to answer the question is something like that: - How to install this thing? - Do not install it or try installing it tomorrow. – Nov 28 '22 at 08:37

Log4j logging directly to elasticsearch server

4 Answers4

Linked