75

I installed Kubernetes on my Ubuntu machine. For some debugging purposes I need to look at the kubelet log file (if there is any such file).

I have looked in /var/logs but I couldn't find a such file. Where could that be?

grizzthedj
  • 7,131
  • 16
  • 42
  • 62
Dimuthu
  • 7,946
  • 6
  • 22
  • 37
  • For a Windows node, in case someone stumbles on this question, they are found in `C:\k` you'll see `kubelet.out.log` and `kubelet.err.log`; that's depending on how you set up your node. – ProfNandaa Jan 27 '23 at 11:27

5 Answers5

146

If you run kubelet using systemd, then you could use the following method to see kubelet's logs:

# journalctl -u kubelet
flyer
  • 9,280
  • 11
  • 46
  • 62
  • 4
    This is the case for Amazon EKS - just found kubelet logs thanks to this comment – lexsys Jun 26 '18 at 10:10
  • 2
    It is also true for Microsoft's AKS. – ewramner Oct 11 '19 at 09:06
  • Works for GKE too, but you might have to set password for your VM user and add it to `systemd-journal` group. Set password using `sudo passwd` and add yourself to the group using `sudo gpasswd -a systemd-journal`, logout and login and the above command should work. – vadasambar Jun 19 '20 at 12:14
  • How can i store this log in specific PATH ? I want to ship my Kubelet logs to ELK stack – Mohammad Ravanbakhsh Feb 20 '23 at 06:09
7

It depends how it was installed. I installed Kubernetes on some Ubuntu machines following the Docker-MultiNode instructions.

With this install, I find the logs using the logs command like this.

  1. Find your container ID.

    $ docker ps | egrep kubelet

  2. Use that container ID to view the logs

    $ docker logs `<container-id>`
Wytrzymały Wiktor
  • 11,492
  • 5
  • 29
  • 37
Rachel
  • 2,858
  • 2
  • 26
  • 30
  • I installed Kubernetes by running kube-up.sh script in cluster directory and I have a single node running as both master and minion – Dimuthu Jan 23 '16 at 00:58
  • 8
    The kubelet is normally a process in the host, not a pod. – scc Jun 01 '18 at 07:12
7

If you are trying to go directly to the file you can find the kubelet logs in /var/log/syslog directory. This is for ubuntu 16.04 and above.

Jonathan_M
  • 111
  • 1
  • 6
4

Finally I could find it in /var/log/upstart directory. Kubernetes in my machine is started using upstart. That's why those log files are in upstart directory

Dimuthu
  • 7,946
  • 6
  • 22
  • 37
1

I installed Kubernetes by kind (Kubernetes in docker).

  1. find docker container of kind to enter
$ docker container ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                 PORTS                                                          NAMES
62588e4d284b        kindest/node:v1.17.0          "/usr/local/bin/entr…"   2 weeks ago         Up 2 weeks             127.0.0.1:32769->6443/tcp                                      kind2-control-plane

$ docker container exec -it kind2-control-plane bash
root@kind2-control-plane:/#

  1. Inside container kind2-control-plane, you could find logfiles in two place:

    • /var/log/containers/
    • /var/log/pods/

And then,you will find they are the same, you can see the example below:

root@kind2-control-plane:/# cat /var/log/containers/redis-master-7db7f6579f-scw95_default_master-f6374281c2c6afcfcd0ee1214d9bd51c1684c0b6c0ba1056295246ecd055563c.log | tail -n 5
2020-04-08T12:09:29.824252114Z stdout F 
2020-04-08T12:09:29.824372278Z stdout F [1] 08 Apr 12:09:29.822 # Server started, Redis version 2.8.19
2020-04-08T12:09:29.824440661Z stdout F [1] 08 Apr 12:09:29.823 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
2020-04-08T12:09:29.824459317Z stdout F [1] 08 Apr 12:09:29.823 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
2020-04-08T12:09:29.82446451Z stdout F [1] 08 Apr 12:09:29.824 * The server is now ready to accept connections on port 6379

root@kind2-control-plane:/# cat /var/log/pods/default_redis-master-7db7f6579f-scw95_094824e1-25aa-4e1e-ab23-d4bae861988a/master/0.log  | tail -n 5
2020-04-08T12:09:29.824252114Z stdout F 
2020-04-08T12:09:29.824372278Z stdout F [1] 08 Apr 12:09:29.822 # Server started, Redis version 2.8.19
2020-04-08T12:09:29.824440661Z stdout F [1] 08 Apr 12:09:29.823 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
2020-04-08T12:09:29.824459317Z stdout F [1] 08 Apr 12:09:29.823 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
2020-04-08T12:09:29.82446451Z stdout F [1] 08 Apr 12:09:29.824 * The server is now ready to accept connections on port 6379

root@kind2-control-plane:/# ls -l /var/log/containers/ | grep redis
lrwxrwxrwx 1 root root 101 Apr  8 12:09 redis-master-7db7f6579f-scw95_default_master-f6374281c2c6afcfcd0ee1214d9bd51c1684c0b6c0ba1056295246ecd055563c.log -> /var/log/pods/default_redis-master-7db7f6579f-scw95_094824e1-25aa-4e1e-ab23-d4bae861988a/master/0.log

If you want to know more in detail about the directories, you can see 2019-2-merge-request in Github.

Jess Chen
  • 3,136
  • 1
  • 26
  • 35