Can logstash read directly from remote log?

Question

I am new to logstash and I am reading about it from couple of days. Like most of the people, I am trying to have a centralized logging system and store data in elasticsearch and later use kibana to visualize the data. My application is deployed in many servers and hence I need to fetch logs from all those servers. Installing logstash forwarder in all those machines and configuring them seems to be a very tedious task (I will do it if this is the only way). Is there a way for logstash to access those logs by mentioning the URL to logs somewhere in conf file instead of logstash forwarders forwarding it to logstash? FYI, my application is deployed on tomcat and the logs are accessible via URL http://:8080/application_name/error.log.

Answer 1

Not directly but there are a few close workarounds - the idea is to create a program/script that will use curl (or it's equivalent) to effectively perform a "tail -f" of the remote log file, and the run that output into logstash.

Here's a bash script that does the trick:

url-tail.sh

This bash script monitors url for changes and print its tail into standard output. It acts as "tail -f" linux command. It can be helpful for tailing logs that are accessible by http.

https://github.com/maksim07/url-tail

Another similar one is here:

https://gist.github.com/habibutsu/5420781

There are others out there, written in php or java: Tail a text file on a web server via HTTP

Once you have that running the question is how to get it into logstash - you could:

1. Pipe it into stdin and use the stdin input

2. Have it append to a file and use the file input

3. Use the pipe input to run the command itself and read from the stdout of the command

The devil is in the details thought, particularly with logstash, so you'll need to experiment but this approach should work for you.