1

I have started a very complex guide on how to install EFK (Elasticsearch + Fluentd + Kibana) on my Kubernetes cluster on DigitalOcean. This process spawned a namespace with 3 different Elasticsearch Pods, 3 Fluentd Pods and 1 Kibana Pod.

CPU jumped from 5% to 95% - permanent. RAM jumped from 34% to about 80% - permanent.

I didn't stop, and continued to trying get the water from the rock - I have forwarded a port so I can checkout the Kibana dashboard, who asked me to provide an index name. I tried to enter logstash-* as described in many articles, but seems that Kibana didn't accept this input so I picked something from the list and no logs did show up.

Eventually I gave up, after 5 hours, and tried to delete the namespace - so I can clean up those actions. But the namespace remain on status: "Terminating" - as long as for 3 hours now.

I just have a very simple Node.js app and I want to see its logs:

  1. Date and time
  2. If its an error I want to see the stack trace.
  3. From node who produced the log.
  4. It will be just amazing to also have the current state (cpu and ram) of the whole system.
Emil Kucharczyk
  • 124
  • 2
  • 12
Raz Buchnik
  • 7,753
  • 14
  • 53
  • 96
  • How did you deploy EFK? Collecting logs, that's one of the best pick. With a web UI, opensource / no subscription required, you won't have lots of options. You don't need 3 ES nodes though, one should be enough to evaluate the solution. Namespace in terminating: my bet is that you've used the ElasticSearch operator. You terminated it. And now you have an ElasticSearch object in your namespace, that kubernetes controllers won't delete by themselves. Tell us how you deployed EFK, so we can figure out what should be done. – SYN Jan 30 '22 at 14:14
  • Hey, just followed this guide, line by line: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes – Raz Buchnik Jan 30 '22 at 14:16
  • Ok, so no operator ... Then I would check for objects left in that namespace, `kubectl get sa,rolebinding,secrets,pvc,all -n kube-logging`. Maybe investigate events as well, `kubectl get events -A -w`. Maybe something specific to DO: try to `kubectl get crd`: for each crd, `kubectl get -n kube-logging`. Tell us what you'll find. If nothing stands out, we may have to check for kube-controller-manager pods logs in kube-system – SYN Jan 30 '22 at 14:22
  • When I make: `kubectl get sa,rolebinding,secrets,pvc,all -n kube-logging` I get: No resources found in kube-logging namespace. – Raz Buchnik Jan 30 '22 at 18:34
  • Still researching. I have noticed that in GKE there is a built in logger focuses on any deployment you desire. – Raz Buchnik Feb 04 '22 at 06:21
  • What version of k8s are you using? Following the guide you mentioned, you had to check if the resources were created at each step. Were those resources created? As a result, you´ve got no resources. So, please provide some logs as SYN asked, ´kubectl cluster-info dump´ output; kube-controller-manager logs. Please let me know the results of your research. – mozello Feb 04 '22 at 09:48

2 Answers2

0

Namespace gets stuck in terminating status when kubernetes cannot delete some of the resources in the namespace. One of useful article for solving stuck in termination is here.

I did not understand fully understand what is your intention.

If you are looking for a centralized solution for logs from all pods (not just one namesapce) and not looking for any paid solution like Datadog or sumologic, ELK stack is one of the best pick. Adding a daemonset is costly since fluentd pod will be in every node and it does not make much sense if your solution is just for one application.

If you are just troubleshooting your app and you do not want to use kubectl logs -l <your app label>, you can use lens or octant to look at logs.

ffran09
  • 887
  • 7
  • 9
0

Eventually I gave up, after 5 hours, and tried to delete the namespace

  • so I can clean up those actions. But the namespace remain on status: "Terminating" - as long as for 3 hours now.

For this namespace issue, you can follow the easy solution just need to remove single line from YAML config and save config : https://stackoverflow.com/a/57726448/5525824

If you just want to debug the application and need logs you can use the kubectl get logs <PODs name>

However, if you are looking for a good solution that worked well with us is using the Graylog with UDP gelf method.

Graylog also uses in background Elasticsearch and MongoDB however there is no collector required at node level instead your application pushes logs to Graylog using Gelf UDP method so not much memory consumption will be there.

Read more at : https://www.graylog.org/

Helm chart : https://github.com/helm/charts/tree/master/stable/graylog

What is GELF UDP ?

The Graylog Extended Log Format (GELF) is a uniquely convenient log format created to deal with all the shortcomings of classic plain Syslog. This enterprise feature allows you to collect structured events from anywhere, and then compress and chunk them in the blink of an eye.

Here is NPM library to push the logs to Graylog : https://www.npmjs.com/package/node-gelf

Harsh Manvar
  • 27,020
  • 6
  • 48
  • 102