Questions tagged [logstash]

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them and send them to storage for later use (such as searching).

logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.

Its logo is a log with a moustache.

Logstash provides many APIs for collecting, parsing, and outputting logs. You can also create your personal API using the Ruby language.

Logstash is often mentioned as one of the components of the (, and ) and is maintained by Elastic.

Useful links:

8605 questions
121
votes
6 answers

Redis Vs RabbitMQ as a data broker/messaging system in between Logstash and elasticsearch

We are defining an architecture to collect log information by Logstash shippers which are installed in various machines and index the data in one elasticsearch server centrally and use Kibana as the graphical layer. We need a reliable messaging…
Salindaw
  • 1,556
  • 2
  • 11
  • 10
101
votes
3 answers

How to handle multiple heterogeneous inputs with Logstash?

Let's say you have 2 very different types of logs such as technical and business logs and you want: raw technical logs be routed towards a graylog2 server using a gelf output, json business logs be stored into an elasticsearch cluster using the…
David
  • 2,603
  • 4
  • 18
  • 28
96
votes
14 answers

How to force Logstash to reparse a file?

I installed Logstash to parse apache files. It took me quite q while to get the settings right and I always tried on real logs. I noticed (as the documentation says) that logstash "remembers" where it was in a file. Now my setings are Ok and I would…
Christophe Claude
  • 1,071
  • 1
  • 8
  • 5
77
votes
7 answers

How to retrieve unique count of a field using Kibana + Elastic Search

Is it possible to query for a distinct/unique count of a field using Kibana? I am using elastic search as my backend to Kibana. If so, what is the syntax of the query? Heres a link to the Kibana interface I would like to make my query:…
Afsheen Khosravian
  • 971
  • 2
  • 8
  • 13
69
votes
4 answers

Export to csv/excel from kibana

I am building a proof of concept using Elasticsearch Logstash and Kibana for one of my projects. I have the dashboard with the graphs working without any issue. One of the requirements for my project is the ability to download the file(csv/excel).…
jsp
  • 2,546
  • 5
  • 36
  • 63
64
votes
4 answers

How to integrate ElasticSearch with MySQL?

In one of my project, I am planning to use ElasticSearch with MySQL. I have successfully installed ElasticSearch. I am able to manage index in ES separately. but I don't know how to implement the same with MySQL. I have read a couple of documents…
Yaxita Shah
  • 1,206
  • 1
  • 11
  • 17
63
votes
5 answers

How to do "where not exists" type filtering in Kibana/ELK?

I am using ELK to create dashboards from my log files. I have a log file with entries that contain an id value and a "success"/"failure" value, displaying whether an operation with a given id succeeded or failed. Each operation/id can fail an…
Anton Kupias
  • 3,945
  • 3
  • 16
  • 20
56
votes
4 answers

logstash check if field exists

I have log files coming in to an ELK stack. I want to copy a field (foo) in order to perform various mutations on it, However the field (foo) isn't always present. If foo doesn't exist, then bar still gets created, but is assigned the literal…
spuder
  • 17,437
  • 19
  • 87
  • 153
53
votes
4 answers

How to handle non-matching Logstash grok filters

I am wondering what the best approach to take with my Logstash Grok filters. I have some filters that are for specific log entries, and won't apply to all entries. The ones that don't apply always generate _grokparsefailure tags. For example, I…
Spanky
  • 5,608
  • 10
  • 39
  • 45
45
votes
2 answers

What are the main differences between Graylog2 and Kibana

What are the main differences between Graylog2 and Kibana? We already use Graylog2 but I must admit I don't really like the UI. Just wonder in case it may be helpful to switch to Kibana.
Sebastien Lorber
  • 89,644
  • 67
  • 288
  • 419
42
votes
1 answer

How to add a numeric filter on kibana dashboard?

I have a field that contains numbers. I want a filter that shows all logs that are less than a constant value. When I try to add a new query filter, all I can see is a query string option.
user1742188
  • 4,563
  • 8
  • 35
  • 60
39
votes
3 answers

Sync postgreSql data with ElasticSearch

Ultimately I want to have a scalable search solution for the data in PostgreSql. My finding points me towards using Logstash to ship write events from Postgres to ElasticSearch, however I have not found a usable solution. The soluions I have found…
Khanetor
  • 11,595
  • 8
  • 40
  • 76
39
votes
10 answers

Removing old indices in elasticsearch

I have the many of my logs indexed in logstash-Year-Week format. That is if i want to delete indices older than a few weeks, how can I achieve that in elasticsearch. Is there an easy, seamless way to do that?
steven johns
  • 477
  • 2
  • 6
  • 8
39
votes
3 answers

Using JSON with LogStash

I'm going out of my mind here. I have an app that writes logs to a file. Each log entry is a JSON object. An example of my .json file looks like the following: {"Property 1":"value A","Property 2":"value B"} {"Property 1":"value x","Property…
user70192
  • 13,786
  • 51
  • 160
  • 240
38
votes
5 answers

Which Serilog sink to use for sending to Logstash?

We started using Serilog in combination with Elasticsearch, and it's a very efficient way to store structure log data (and later visualize them using tools like Kibana). However, I see the advantage of not writing log data directly to the backend…
Vagif Abilov
  • 9,835
  • 8
  • 55
  • 100
1
2 3
99 100