Logstash File Input stream events from Files
Questions tagged [logstash-file]
221 questions
13
votes
1 answer
How to decrease Logstash Memory Usage
I am using Logstash-5.6.5 (in Windows) running in a standalone system (no cloud or cluster). Planning to watch some log files and post it to locally run elasticsearch. But when checked the Logstash's memory usage, without a configuration to watch…
Loganathan
- 903
- 2
- 10
- 23
13
votes
2 answers
Logstash close file descriptors?
BACKGROUND:
We have rsyslog creating log files directories like: /var/log/rsyslog/SERVER-NAME/LOG-DATE/LOG-FILE-NAME
So multiple servers are spilling out their logs of different dates to a central location.
Now to read these logs and store them in…
Siddharth Trikha
- 2,648
- 8
- 57
- 101
10
votes
2 answers
Java Filter For Logstash
You know how there is a Ruby filter for Logstash which enables me to write code in Ruby and it is usually included in the config file as follows
filter {
ruby {
code => "...."
}
}
Now I have two Jar files that I would like to…
hello_its_me
- 743
- 2
- 19
- 52
9
votes
2 answers
Delete logs after consumption: logstash
I have logstash watching a directory on the host. Every time it sees a log that matches the path I specified in the logstash config it starts to import the data into my elasticsearch cluster. Does logstash have a way to delete the log after it is…
tomer
- 231
- 2
- 4
- 12
7
votes
0 answers
Configure logstash to read logs from Amazon S3 bucket
I have been trying to configure logstash to read logs which are getting generated in my amazon S3 bucket, but have not been successful. Below are the details :
I have installed logstash on an ec2 instance
My logs are all gz files in the s3…
user1697058
- 115
- 1
- 9
6
votes
1 answer
How to map nested JSON in Log-stash HTTP Output
I am using Logstash to output JSON message to an API. I am using "mapping" attribute to map my message. See, following piece of my shipper configurations.
output {
stdout { }
http {
url => "http://localhost:8087/messages"
…
omer khalid
- 855
- 1
- 12
- 39
6
votes
1 answer
Logstash: Handling of large messages
I'm trying to parse a large message with Logstash using a file input, a json filter, and an elasticsearch output. 99% of the time this works fine, but when one of my log messages is too large, I get JSON parse errors, as the initial message is…
praddc
- 103
- 1
- 6
5
votes
1 answer
Logstash File input: sincedb_path
Upon restarting Logstash, at times observed that Logstash duplicates the log events. Was wondering as to what would be the right way to apply start_position, sincedb_path, sincedb_write_interval configuration options.
What happens when there are…
John C
- 1,795
- 4
- 27
- 42
4
votes
1 answer
retry_non_idempotent in Logstash Http output
I am using the default settings [1] with the following config
http {
format=>"message"
http_method=>"post"
url=>"xxx"
message=>'xxx'
}
Which
retry_failed = true
automatic_retries =…
Ryan
- 10,041
- 27
- 91
- 156
4
votes
1 answer
Log rotation in logstash
I am using file as input for logs in logstash . My log files are rotated daily so , I wanted to ask how can we configure file plugin of logstash so that it work with the files that are rotated daily. And adding to this, is log rotation available…
rresol
- 323
- 4
- 20
3
votes
0 answers
Empty lines in multiline pattern(python error traceback) in filebeat input are not getting parsed correctly?
The log line which should be harvested and published to logstash as a single line:
[pid: 17318|app: 0|req: 1/2] 10.14.206.28 (jaavedkhan) {60 vars in 1296 bytes} [Mon Dec 30 15:51:38 2019] GET /en/ => generated 27 bytes in 711 msecs (HTTP/1.1 500) 6…
Javed
- 5,904
- 4
- 46
- 71
3
votes
1 answer
Reading from rotating log files in logstash
As per the documentation of logstash's file plugin, the section on File Rotation says the following:
To support programs that write to the rotated file for some time after
the rotation has taken place, include both the original filename and
the…
Wand Maker
- 18,476
- 8
- 53
- 87
3
votes
2 answers
logstash simple file input/output
I have trouble getting logstash to work. The Basic logstash Example works. But then I struggle with the Advanced Pipeline Example. Perhaps it could be as well a problem with elasticsearch.
Now I just want to check if a simple example work:
input:…
jerik
- 5,714
- 8
- 41
- 80
2
votes
2 answers
Add documents to elasticsearch if it does not exists
I transferred some data from a log generated every day to elasticsearch using logstash, and my logstash output section looks like :
i keep the same id (id_ot) in both my log file and elasticsearch, but what i would like to do is : if the new coming…
Mohamed
- 239
- 1
- 4
- 17
2
votes
0 answers
How to add a tag when messages is multiline in Logstash
I use Filebeat6x to ship my logs to logstash.
Some of my logs may be a multiline thats why I use Filebeat to Manage multiline messages
Now I want to add filter in logstash to do something like
if the message is multiline then add tag.
If the…
airdata
- 577
- 1
- 7
- 23