Stack Overflow
Stack Overflow

Questions tagged [filebeat]

Filebeat is a lightweight, open source shipper for log file data. As the next-generation Logstash Forwarder, Filebeat tails logs and quickly sends this information to Logstash for further parsing and enrichment or to Elasticsearch for centralized storage and analysis.

Filebeat is a lightweight, open source shipper for log file data. As the next-generation Logstash Forwarder, Filebeat tails logs and quickly sends this information to Logstash for further parsing and enrichment or to Elasticsearch for centralized storage and analysis. See more details

1457 questions
19
votes
4 answers

check if a string starts with number using regular expression

I am writing a filebeat configuration when I am matching if a line starts with a number like 03:32:33 ( a timestamp). I am currently doing it by- \d But its not getting recognised, is there anything else which I should do. I am not particularly…
Y0gesh Gupta
  • 2,184
  • 5
  • 40
  • 56
14
votes
1 answer

Elasticsearch: No handler for type [keyword] declared on field [hostname]

I get above Mapper Parsing Error on Elasticsearch when indexing log from filebeat. I tried both Filebeat -> Elasticserach and Filebeat -> Logstash -> Elasticsearch approach. I have followed their own documentations, I installed filebeat template as…
rayhan
  • 636
  • 3
  • 9
  • 26
13
votes
1 answer

Filebeat - parse fields from message line

I am using Filebeat to ship log data from my local txt files into Elasticsearch, and I want to add some fields from the message line to the event - like timestamp and log level. For example here is one of my log lines: 2016-09-22 13:51:02,877 INFO…
Hadash
  • 228
  • 1
  • 2
  • 7
12
votes
4 answers

Difference between using Filebeat and Logstash to push log file to Elasticsearch

I am trying out the ELK to visualise my log file. I have tried different setups: Logstash file input plugin https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html Logstash Beats input plugin…
user1589188
  • 5,316
  • 17
  • 67
  • 130
12
votes
2 answers

Filebeat vs Rsyslog for forwarding logs

I am currently using filebeat to forward logs to logstash and then to elasticsearch. Now, I am thinking about forwarding logs by rsyslog to logstash. The benefit of this would be that, I would not need to install and configure filebeat on every…
Luv33preet
  • 1,686
  • 7
  • 33
  • 66
12
votes
1 answer

What is the point of REDIS in ELK stack?

I currently have architecture with filebeat as the log shipper, which sends logs to log stash indexer instance and then to managed elastic search in AWS. Due to persistent TCP connections, I cannot load balance using AWS ELB multiple log stash…
alexfvolk
  • 1,810
  • 4
  • 20
  • 40
11
votes
2 answers

Logstash with persistent queue

I have started logstash using following configurations: Inside logstash.yml: queue.type: persisted queue.max_bytes: 8gb queue.checkpoint.writes: 1 configuration file: input { beats { port => "5043" } } filter { grok { …
Abesalomi Gogatishvili
  • 193
  • 1
  • 2
  • 8
11
votes
1 answer

Running Filebeat in windows

I set up filebeat on windows recently using these instructions https://www.elastic.co/downloads/beats/filebeat but it forces me to keep a cmd prompt open running the command filebeat.exe -c filebeat.yml I am wondering if there is a way to run this…
ScipioAfricanus
  • 1,331
  • 6
  • 18
  • 39
11
votes
2 answers

FileBeat directly to ELS or via LogStash?

We are installing ELS and Kibana for log aggregation/analysis. The first system to use it is greenfield so we output structured logs from the services that make up our system. Given that we don't need to add structure to our logs I was planning on…
Myles McDonnell
  • 12,943
  • 17
  • 66
  • 116
10
votes
1 answer

Can Filebeat use multiple config files?

I have several applications running on a single server. I'd like to use filebeat to ship the logs of each of them to logstash. However, for the sake of configuration management, I'd like to be able to add configuration to filebeat for each app…
izrik
  • 918
  • 2
  • 9
  • 20
10
votes
1 answer

Kafka-Connect vs Filebeat & Logstash

I'm looking to consume from Kafka and save data into Hadoop and Elasticsearch. I've seen 2 ways of doing this currently: using Filebeat to consume from Kafka and send it to ES and using Kafka-Connect framework. There is a Kafka-Connect-HDFS and…
Adrian
  • 5,603
  • 8
  • 53
  • 85
9
votes
1 answer

Windows docker: permission denied /var/run/docker.sock

When I try to run filebeat with autodiscover I get the following error: Exiting: error in autodiscover provider settings: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get …
cdalxndr
  • 1,435
  • 1
  • 15
  • 20
9
votes
3 answers

How do I force rebuild log's data in filebeat 5

I have filebeats 5.x ship logs to logstash. How do I reset the “file pointer” in filebeat This is a similar problem to How to force Logstash to reparse a file? https://discuss.elastic.co/t/how-do-i-reset-the-file-pointer-in-filebeats/49440 I…
周宏成
  • 175
  • 1
  • 1
  • 8
8
votes
3 answers

Filebeat : data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path

Looking at the logs in one of the filebeat pods i can see this: 2021-01-04T10:10:52.754Z DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:129 add_cloud_metadata: fetchMetadata ran for 2.351101ms 2021-01-04T10:10:52.754Z …
user63898
  • 29,839
  • 85
  • 272
  • 514
8
votes
3 answers

logstash: how to include input file line number

I am trying to create a way to navigate my log files and the main features I need are: search for strings inside log file (and returning line of occurrences). pagination from line x to line y. Now I was checking Logstash and it was looking great…
eLRuLL
  • 18,488
  • 9
  • 73
  • 99
1
2 3
…
97 98