7

I read a blog post about setting up Logstash, Elasticsearch and Kibana, and the author suggested to use NXLog to ship logs from different machines. A typical distributed scenario presented in "The Logstash Book" shows how Logstash can be used in both shipping and indexing role. We are currently experimenting with Logstash and setting up it to ship logs to Elasticsearch was straighforward. So I wonder why people choose NXLog as a log shipper to Logstash instead of using Logstash on both ends.

Vagif Abilov
  • 9,835
  • 8
  • 55
  • 100
  • It's just another useful tool for people who have very common logging infrastructure. You can write your own process if you find that simpler. – coffeeaddict Sep 05 '14 at 15:29

3 Answers3

18

The logstash-forwarder project, formerly known as "Lumberjack", explains it like so:

Resource Usage Concerns

Perceived Problems: Some users view logstash releases as "large" or have a generalized fear of Java.

Actual Problems: Logstash, for right now, runs with a footprint that is not friendly to underprovisioned systems such as EC2 micro instances; on other systems it is fine. This project will exist until that is resolved.

Transport Problems

Few log transport mechanisms provide security, low latency, and reliability.

The lumberjack protocol used by this project exists to provide a network protocol for transmission that is secure, low latency, low resource usage, and reliable.

A Logstash shipper instance isn't especially heavyweight, but if your machine only has 1-2GB of RAM it can be hard to casually allocate several hundred MB to another JVM instance.

Another consideration: what if your shipper nodes are running an OS that isn't supported by Logstash? Logstash can run on Windows, now, but it's still on the buggy side of things. I can't specifically vouch for NXLog in that area, but I gather it's a popular choice.

rutter
  • 11,242
  • 1
  • 30
  • 46
  • 1
    1) Avoid installing java (which java? which version?) on lots of machines where we wouldn't otherwise have it. 2) nxlog can read Windows Event Log nicely, and can ship over as JSON over TLS (so we're not tying ourselves too heavily to nxlog). – Cameron Kerr Oct 29 '14 at 21:21
  • But what advantage does NXLog have over the lightweight "Logstash Forwarder" agent that Logstash provides? – Mark Dec 15 '14 at 14:18
2

We've implemented shipping with RSyslog rather than LogStash due to a desire to keep java off of our hosts where possible. It makes the LogStash config on the loghost/filer a little more complex, but not having to keep up with Java security upgrades on front-end hosts is more of a pain than dealing with ripping logs apart on the central filer.

Gary Rogers
  • 395
  • 2
  • 7
0

I use nxlog as it is source, platform and destination agnostic.

As a security team we need lots of data from everywhere but don't want to also be responsible for handling operational data. Retrieving data after it has been delivered to logstash or splunk is not viable either. So nxlog allows us to have our cake and eat it: we send data to the security collection infrastructure and allow operational teams to send some or all of that data (or even data we aren't interested in) wherever they want.

It also fulfills the core requirement of having infrastructure as replaceable components... should something better come along, we can replace individual components (e.g. replace storm with flink) without having to change the entire infrastructure (as we did moving away from splunk)

Djarid
  • 446
  • 4
  • 11