2

I have deployed EFK stack on Kubernetes and I get following error messages and looks like Fluentd cannot communicate with elasticsearch.

[warn]: [elasticsearch] Could not communicate to Elasticsearch, resetting connection and trying again. getaddrinfo: Temporary failure in name resolution (SocketError)

[warn]: [elasticsearch] Remaining retry: 11. Retry to communicate after 16 second(s)

Deploying different Helm charts makes any difference!!? Deployed kiwigrid/fluentd-elasticsearch for Fluentd and bitnami/elasticsearch for elasticsearch and stable/kibana for Kibana. However, Ping or SSH to fluentd is possible from my external Linux machine. EFK stack runs inside same namespace. Rancher tool monitors whole cluster therefore I see my Fluentd logs. Loadbalancer IP for Kibana and Elasticsearch is set. Able to see Kibana UI page.

elasticsearch-elasticsearch-master-1 1/1 Running 0 24h fluentd-elasticsearch-4q6cx 1/1 Running 3 47m

Anybody who can support in this problem? pretty new to K8s.

kishorK
  • 453
  • 2
  • 7
  • 16
  • Are you able to `ping` or `ssh` to Elasticsearch's machine? – Azeem Jul 31 '20 at 10:06
  • @Azeem Yes, I can go inside container from my local Linux machine and see some files: "kubectl exec -it elasticsearch-elasticsearch-master-1 -n logging -- bash" – kishorK Jul 31 '20 at 11:32
  • But, are you able to ping it from an external machine (preferably, where fluentd resides)? I'm assuming that the container's IP has already been exposed. – Azeem Jul 31 '20 at 11:35
  • Yes, I am using loadbalancer IP to expose to end-user. and EFK stack resides on same namespace. – kishorK Jul 31 '20 at 11:37
  • Right. Did you try to send logs through some command-line tool for testing? – Azeem Jul 31 '20 at 11:46
  • I see logs of elasticsearch container in Rancher tool where I can monitor whole cluster – kishorK Jul 31 '20 at 11:49
  • How about manually sending a log e.g. using `curl`? – Azeem Jul 31 '20 at 11:51
  • 1
    Okay. You should add this information in your question as well. Those errors show that it's a connectivity issue but you've already verified that it's working for other stuff. I'd suggest running fluentd with`-v` or `-vv` flag and observe debug issues. If the connectivity issue still persists even after multiple retries, open an issue for this plugin on fluentd GitHub repo. – Azeem Jul 31 '20 at 11:59
  • @Azeem Thank you for valuable suggestions. I run fluentd -v and -vv but still same error messages which I mentioned inside question. – kishorK Jul 31 '20 at 12:40

0 Answers0