Logstash configuration has three main sections - Input, Filter and Output. Each section has configurations for each plugin available in that section.
Questions tagged [logstash-configuration]
1241 questions
56
votes
4 answers
logstash check if field exists
I have log files coming in to an ELK stack. I want to copy a field (foo) in order to perform various mutations on it, However the field (foo) isn't always present.
If foo doesn't exist, then bar still gets created, but is assigned the literal…
spuder
- 17,437
- 19
- 87
- 153
21
votes
7 answers
Logstash configtest
I ran service logstash configtest but error given was:
logstash: unrecognized service
I was able to run logstash service individually but not with "configtest". In etc/logstash/conf.d/ I created logstash.conf file where consist of code as present…
Shann
- 311
- 1
- 2
- 9
19
votes
1 answer
Logstash - remove deep field from json file
I have JSON file that I'm sending to ES through logstash. I would like to remove 1 field ( It's deep field ) in the JSON - ONLY if the value is NULL.
Part of the JSON is:
"input": {
"startDate": "2015-05-27",
"numberOfGuests": 1,
…
Amit Daniel
- 297
- 1
- 5
- 16
17
votes
3 answers
multiple inputs on logstash jdbc
I am using logstash jdbc to keep the things syncd between mysql and elasticsearch. Its working fine for one table. But now I want to do it for multiple tables. Do I need to open multiple in terminal
logstash agent -f…
Asim Zaidi
- 27,016
- 49
- 132
- 221
16
votes
1 answer
Sending Docker container logs to ELK Stack by configuring the logging drivers - Easy Method
I usually run applications as docker containers because of its high flexibility and availability. Is there a way to get the container logs into my logstash server.
Kishor Unnikrishnan
- 1,928
- 4
- 21
- 33
13
votes
1 answer
How to decrease Logstash Memory Usage
I am using Logstash-5.6.5 (in Windows) running in a standalone system (no cloud or cluster). Planning to watch some log files and post it to locally run elasticsearch. But when checked the Logstash's memory usage, without a configuration to watch…
Loganathan
- 903
- 2
- 10
- 23
13
votes
1 answer
Logstash config, "if string contains..."
So, let's assume that I have a portion of a log line that looks something like this:
GET /restAPI/callMethod1/8675309
The GET matches a http method, and get's extracted, the remainder matches a URI, and also gets extracted. Now in the logstash…
A_Elric
- 3,508
- 13
- 52
- 85
13
votes
2 answers
Logstash filter remove_field for all fields except a specified list of fields
I am parsing a set of data into an ELK stack for some non-tech folks to view. As part of this, I want to remove all fields except a specific known subset of fields from the events before sending into ElasticSearch.
I can explicitly specify each…
redstonemercury
- 354
- 1
- 2
- 9
12
votes
3 answers
Logstash SQL Server Data Import
input {
jdbc {
jdbc_driver_library => "sqljdbc4.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://192.168.2.126\\SQLEXPRESS2014:1433;databaseName=test
…
Vivek Gupta
- 349
- 1
- 2
- 15
12
votes
1 answer
Block Comments in Logstash Config file
I am just getting started with Logstash and wondering whether there is a way to specify block comments in logstash config file?
It will be very useful as I am testing with long grok patterns which span multiple lines.
Yellowjacket
- 548
- 2
- 7
- 19
11
votes
2 answers
Logstash start with error: Pipelines YAML file is empty
When I try start logstash server on my machine I get this error:
Sending Logstash's logs to D:/kibana/logstash-6.3.2/logs which is now configured
via log4j2.properties
ERROR: Pipelines YAML file is empty. Location:…
ip696
- 6,574
- 12
- 65
- 128
11
votes
2 answers
Logstash with persistent queue
I have started logstash using following configurations:
Inside logstash.yml:
queue.type: persisted
queue.max_bytes: 8gb
queue.checkpoint.writes: 1
configuration file:
input {
beats {
port => "5043"
}
}
filter {
grok {
…
Abesalomi Gogatishvili
- 193
- 1
- 2
- 8
11
votes
3 answers
Sending data to logstash via tcp
I'm running into some issues sending log data to my logstash instance from a simple java application. For my use case, I'm trying to avoid using log4j logback and instead batch json events on separate lines through a raw tcp socket. The reason for…
user1553248
- 1,184
- 2
- 19
- 33
10
votes
1 answer
Is it possible to launch a Ruby debugger from within the Logstash Ruby filter plugin?
Is it possible to launch a Ruby debugger from within the Logstash Ruby filter plugin? It would be very handy for debugging.
David P
- 3,604
- 3
- 37
- 54
10
votes
2 answers
Java Filter For Logstash
You know how there is a Ruby filter for Logstash which enables me to write code in Ruby and it is usually included in the config file as follows
filter {
ruby {
code => "...."
}
}
Now I have two Jar files that I would like to…
hello_its_me
- 743
- 2
- 19
- 52