1

I have a docker swarm cluster and am able to get all docker "container" logs to ELK stack.

But am unable to get docker daemon logs. Can someone please guide me to achieve this.

FYI : My stack is in Linux.

Sunil Agarwal
  • 4,097
  • 5
  • 44
  • 80

2 Answers2

1

You can use Filebeat plugin to send the logs from the daemon logs file to your ELK (plugin presentation page.

There is an article on this point on the elasic.co blog. Your configuration will be different since you don't want containers logs but Docker daemon logs found at the path /var/log/docker.log or /var/log/daemon.log.

EDIT 1:

Since in your environment, the logs are readable with journalctl, I digged up the internet and I have found an ELK plugin that allows you to send the logs from the journald: https://github.com/logstash-plugins/logstash-input-journald

I Hope it'll help.

Paul Rey
  • 1,270
  • 1
  • 15
  • 26
1

1st: you'd need to find out where your docker daemon is saving the logs, which depends on linux distribution. See this response with a list of possible places:

https://stackoverflow.com/a/30970134/3165889

2nd: you can use the suggestion of Paul Rey and use Filebeat. As an alternative, I also suggest the use of Fluentd, which usually you can use in place of Logstash, then having EFK instead of ELK, or simply as an extra tool to your ELK environment.

It can also read from a file using the tail input plugin

It can also insert data to Elasticsearch using the elasticsearch out plugin

This tutorial teaches how to log containers, but then you'd need to change your input plugin to tail from that file: Docker logging via EFK

I'd also like to add that, if you're interested in logging the daemon, you probably want to log even if docker is failing to start. So I'd install Fluentd directly on the host. NOT in a container.

Joao Cunha
  • 772
  • 4
  • 15
  • i can get logs from journalctl but from there how can i continuously push logs to ELK? – Sunil Agarwal Aug 03 '18 at 06:28
  • This means that you probably have systemd logging your process. journalctl is used by systemd but saves logs to binary files. That's why you have to use it to read the logs. Although, If you're lucky and you have a newer version of systemd, you can setup systemd to save logs to a specific file, see: https://stackoverflow.com/a/48052152/3165889. After having it on a file, you can use simply the tail input plugin from fluentd – Joao Cunha Aug 05 '18 at 10:42
  • Another option, which I haven't tested, but looks like someone already wrote a plugin to integrate systemd journal and fluend: https://github.com/reevoo/fluent-plugin-systemd – Joao Cunha Aug 05 '18 at 10:43