Filebeat fails to connect to logstash

Question

I'm using two servers on the cloud on one server (A) I installed filebeat and on second server (B) I have installed logstash, elasticsearch, and kibana. So I'm facing problem while sending logs from server A to server B on logstash.

My filebeat configuration is

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/vinit/demo/*.log
  fields:
    log_type: apache
  fields_under_root: true

#output.elasticsearch:
  #hosts: ["localhost:9200"]
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

output.logstash:
  hosts: ["XXX.XX.X.XXX:5044"]
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  #ssl.certificate: "/etc/pki/client/cert.pem"
  #ssl.key: "/etc/pki/client/cert.key"

In logstash, I have enabled modules system, filebeat, and logstash.

Logstash configuration is

input {
  beats {
    port => 5044
  }
}
filter {
        grok {
                match => { "message" => "^%{IP:CLIENT_IP} (?:-|%{USER:IDEN}) (?:-|%{USER:AUTH}) \[%{HTTPDATE:CREATED_ON}\] \"(?:%{WORD:REQUEST_METHOD} (?:/|%{NOTSPACE:REQUEST})(?: HTT$
                add_field => {
                        "LOG_TYPES" => "apache-log"
                }
        overwrite => [ "message" ]
        }
}
output {
        elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "apache-info-log"
        }
  stdout { codec => rubydebug }
}

In Elasticsearch I did

network.host: localhost

I'm getting error are below-

|2019-01-18T15:05:47.738Z|INFO|crawler/crawler.go:72|Loading Inputs: 1|
|---|---|---|---|
|2019-01-18T15:05:47.739Z|INFO|log/input.go:138|Configured paths: [/home/vinit/demo/*.log]|
|2019-01-18T15:05:47.739Z|INFO|input/input.go:114|Starting input of type: log; ID: 10340820847180584185 |
|2019-01-18T15:05:47.740Z|INFO|log/input.go:138|Configured paths: [/var/log/logstash/logstash-plain*.log]|
|2019-01-18T15:05:47.740Z|INFO|log/input.go:138|Configured paths: [/var/log/logstash/logstash-slowlog-plain*.log]|
|2019-01-18T15:05:47.742Z|INFO|log/harvester.go:254|Harvester started for file: /home/vinit/demo/info-log.log|
|2019-01-18T15:05:47.749Z|INFO|log/input.go:138|Configured paths: [/var/log/auth.log* /var/log/secure*]|
|2019-01-18T15:05:47.763Z|INFO|log/input.go:138|Configured paths: [/var/log/messages* /var/log/syslog*]|
|2019-01-18T15:05:47.763Z|INFO|crawler/crawler.go:106|Loading and starting Inputs completed. Enabled inputs: 1|
|2019-01-18T15:05:47.763Z|INFO|cfgfile/reload.go:150|Config reloader started|
|2019-01-18T15:05:47.777Z|INFO|log/input.go:138|Configured paths: [/var/log/auth.log* /var/log/secure*]|
|2019-01-18T15:05:47.790Z|INFO|log/input.go:138|Configured paths: [/var/log/messages* /var/log/syslog*]|
|2019-01-18T15:05:47.790Z|INFO|input/input.go:114|Starting input of type: log; ID: 15514736912311113705 |
|2019-01-18T15:05:47.790Z|INFO|input/input.go:114|Starting input of type: log; ID: 4004097261679848995 |
|2019-01-18T15:05:47.791Z|INFO|log/input.go:138|Configured paths: [/var/log/logstash/logstash-plain*.log]|
|2019-01-18T15:05:47.791Z|INFO|log/input.go:138|Configured paths: [/var/log/logstash/logstash-slowlog-plain*.log]|
|2019-01-18T15:05:47.791Z|INFO|input/input.go:114|Starting input of type: log; ID: 2251543969305657601 |
|2019-01-18T15:05:47.791Z|INFO|input/input.go:114|Starting input of type: log; ID: 9013300092125558684 |
|2019-01-18T15:05:47.791Z|INFO|cfgfile/reload.go:205|Loading of config files completed.|
|2019-01-18T15:05:47.792Z|INFO|log/harvester.go:254|Harvester started for file: /var/log/secure-20181223|
|2019-01-18T15:05:47.794Z|INFO|log/harvester.go:254|Harvester started for file: /var/log/messages-20181223|
|2019-01-18T15:05:47.797Z|INFO|log/harvester.go:254|Harvester started for file: /var/log/secure-20181230|
|2019-01-18T15:05:47.800Z|INFO|log/harvester.go:254|Harvester started for file: /var/log/messages-20181230|
|2019-01-18T15:05:47.804Z|INFO|log/harvester.go:254|Harvester started for file: /var/log/secure-20190106|
|2019-01-18T15:05:47.804Z|INFO|log/harvester.go:254|Harvester started for file: /var/log/secure|
|2019-01-18T15:05:47.804Z|INFO|log/harvester.go:254|Harvester started for file: /var/log/secure-20190113|
|2019-01-18T15:05:47.816Z|INFO|log/harvester.go:254|Harvester started for file: /var/log/messages-20190106|
|2019-01-18T15:05:47.817Z|INFO|log/harvester.go:254|Harvester started for file: /var/log/messages|
|2019-01-18T15:05:47.818Z|INFO|log/harvester.go:254|Harvester started for file: /var/log/messages-20190113|
|2019-01-18T15:05:47.855Z|INFO|pipeline/output.go:95|Connecting to backoff(async(tcp://XXX.XX.X.XXX:5044))|

|2019-01-18T15:06:18.855Z|ERROR|pipeline/output.go:100|Failed to connect to backoff(async(tcp://XXX.XX.X.XXX:5044)): dial tcp XXX.XX.X.XXX:5044: i/o timeout|
|---|---|---|---|
|2019-01-18T15:06:18.855Z|INFO|pipeline/output.go:93|Attempting to reconnect to backoff(async(tcp://XXX.XX.X.XXX:5044)) with 1 reconnect attempt(s)|

Is anyone have any idea how to resolve this and make it work properly?

See if you can manually connect (telnet, nc) from the FB machine to the LS machine on port 5044. — Alain Collins, Jan 18 '19 at 18:25
@AlainCollins thank you for reply.. well I'm using it on remote server so it's saying telnet command not found. — Vinit Jordan, Jan 18 '19 at 18:52
Is server B configured to accept connection from server A ? On cloud you will need to configure security groups correctly — ben5556, Jan 21 '19 at 09:02
If you don't have telnet installed, use a different tool (e.g. 'nc') or installed it. You need to test the connectivity from the client to the server to rule out a networking/permissions issue. — Alain Collins, Jan 22 '19 at 18:13
I was getting this error because of 5044 port number is not whitelisted. I added this port in the whitelisted group now it's working as I was expecting. — Vinit Jordan, May 14 '19 at 04:38
@Aman Kindly follow these [steps](https://stackoverflow.com/questions/17161345/how-to-open-a-web-server-port-on-ec2-instance?answertab=votes#tab-top) if you want to open port on EC2. — Vinit Jordan, Aug 07 '19 at 06:16
Refer to [Failed to connect to backoff(async(tcp://ip:5044)): dial tcp ip:5044: i/o timeout](https://stackoverflow.com/q/57635755/6521116) — LF00, Aug 26 '19 at 03:02

TaskerBliss · Answer 1 · 2023-03-12T09:53:23.847

Related question is Failed to connect to backoff(async(tcp://ip:5044)): dial tcp ip:5044: i/o timeout. In the answer, it was proposed to allow outgoing TCP connection on port 5044 directly in your cloud provider's settings page, as it may be blocked by default.

In addition to present comments by @Vinit Jordan, who whitelisted port 5044 on EC2 with this steps, I propose possible solution for general case.

Please, check your default firewall on logstash server. Probably you have ufw (simple firewall that was preconfigured during initial Nginx setup). I ran into this problem right after installation of ELK on the machine B and Filebeat on the machine A.

I just added a new rule for Filebeat server ufw firewall and the error disappeared:

sudo ufw allow from <IP_address_of_machine_A> to any port 5044

Then, Filebeat log on machine A showed me:

"message":"Connection to backoff(async(tcp://<IP_address_of_machine_B>:5044)) established"

It is also reasonable to add more general rule for your trusted servers:

sudo ufw allow from <IP_ADDRESS>

Filebeat fails to connect to logstash

1 Answers1