How do you ingest Spring boot logs directly into elastic

Question

I’m investigating feasability of sending spring boot application logs directly into elastic search. Without using filebeats or logstash. I believe the Ingest plugin may help with this.

My initial thoughts are to do this using logback over TCP.

https://github.com/logstash/logstash-logback-encoder

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
      <destination>127.0.0.1:4560</destination>
      <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
  </appender>

  <root level="DEBUG">
      <appender-ref ref="stash" />
  </root>
</configuration>

So looking at the above you can send logs directly into logstash. Im just wondering if it was possible to use the newer functionality of ingest and skip using logstash? By sending json encoded logs directly into elastic over the network using the ingest method?

https://www.elastic.co/blog/new-way-to-ingest-part-1

My question

I’m wondering if this is possible? If so could you explain how you would do it. Also what possible what would be the pitfalls etc.

Answer 1

I just tried my suggestion and it worked out perfectly.

First, add this dependency in your POM:

    <dependency>
        <groupId>org.logback-extensions</groupId>
        <artifactId>logback-ext-loggly</artifactId>
        <version>0.1.2</version>
    </dependency>

Then, in your logback.xml configuration, add an appender and a logger like this:

<appender name="ES" class="ch.qos.logback.ext.loggly.LogglyAppender">
    <endpointUrl>http://localhost:9200/tests/test?pipeline=logback</endpointUrl>
    <pattern>%m</pattern>
</appender>
<logger name="es" level="INFO" additivity="false">
    <appender-ref ref="ES"/>
</logger>

You also need to define an ingest pipeline like this:

PUT _ingest/pipeline/logback
{
  "description": "logback pipeline",
  "processors": [
    {
      "set" : {
        "field": "source",
        "value": "logback"
      }
    }
  ]
}

Then, in your code you can use that logger and send whatever data you have to your ES

private Logger esLogger = LoggerFactory.getLogger("es");
...
esLogger.info("{\"message\": \"Hello World from Logback!\"}");

And this document will end up in your ES:

{
    "_index": "tests",
    "_type": "test",
    "_id": "AV3Psj5MF_PW7ho1yJhQ",
    "_score": 1,
    "_source": {
      "source": "logback",
      "message": "Hello World from Logback!",
    }
}

Answer 2

I asked the same question earlier: Direct integration of Logback with Elasticsearch

Take a look to https://github.com/internetitem/logback-elasticsearch-appender

UPDATE Development of logback-elasticsearch-appender stalled in October 2017 with breaking issues for ES v6.x