Where are the Docker Desktop for Windows kubelet logs located?

Question

I can't find where the kubelet logs are located for Docker Desktop (Windows). There's a similar question here, but the answers all refer to linux/kind installs of kubernetes.

Answer 1

To get kubelet logs you need to get access to the virtual machine that docker daemon runs in. Since there is no ssh available there is workaround for this:

Here`s how to login into VM:

docker run --privileged -it -v /:/host -v /var/run/docker.sock:/var/run/docker.sock jongallant/ubuntu-docker-client

and then use this command to get the kubelet logs:

ls /host/var/log/kubelet*

Please note that this is just workaround for tool that was designed for testing and it`s not the official supported way. This case also describes how to ssh to docker deamon.

Answer 2

I get that this is an old question, but it does seem to get some traffic, has no accepted answer and is the only of this kind on SO - here is my take on this problem and a solution:

The process

As already pointed out by @acid_fuji, everything is running inside a VM we need to access, to get the required information. As most things in Linux have some representation in the filesystem, and tools like ps use procfs to query the requested information, mounting the VMs root directory (/) ist sufficient for introspection and quite easy:

Run a container and mount the VMs root directory / to /host:

docker run --rm -it -v /:/host alpine

In that container, get a shell with the VMs root directory in /host as root /:

chroot /host

From this shell with changed root directory, tools like ps will return information about the VM and not your container anymore. Next step is to find some information about the running kubelet. Get a tree formatted list of all running processes in the VM with their command line and highlight kubelet:

ps -ef --forest | grep --color -E 'kubelet|$'

Reduced to the relevant portions, the result will look something like this:

UID        PID  PPID  C STIME TTY          TIME CMD
...
root        30     1  0 Aug11 ?        00:05:09 /usr/bin/memlogd -fd-log 3 -fd-query 4 -max-lines 5000 -max-line-len 1024
...
root       543     1  0 Aug11 ?        00:00:16 /usr/bin/containerd-shim-runc-v2 -namespace services.linuxkit -id docker -address /run/containerd/containerd.sock
root       563   543  0 Aug11 ?        00:00:02  \_ /usr/bin/docker-init /usr/bin/entrypoint.sh
root       580   563  0 Aug11 ?        00:00:00  |   \_ /bin/sh /usr/bin/entrypoint.sh
root       679   580  0 Aug11 ?        00:00:00  |       \_ /usr/bin/logwrite -n lifecycle-server /usr/bin/lifecycle-server
root       683   679  0 Aug11 ?        00:00:35  |           \_ /usr/bin/lifecycle-server
...
root      1539   683  0 Aug11 ?        00:00:01  |               \_ /usr/bin/logwrite -n kubelet kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --config /etc/kubeadm/kubelet.yaml --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --hostname-override=docker-desktop --container-runtime=remote --container-runtime-endpoint unix:///var/run/cri-dockerd.sock
root      1544  1539  2 Aug11 ?        00:38:38  |                   \_ kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --config /etc/kubeadm/kubelet.yaml --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --hostname-override=docker-desktop --container-runtime=remote --container-runtime-endpoint unix:///var/run/cri-dockerd.sock

We find kubelet as PID 1544, spawned by logwrite (PID 1539). Issuing logwrite --help we find the -n flag to set the name to appear in logs for the launched instance, and logwrite to be sending its logs to memlogd, which we find as PID 30.

Knowing what to search for I found this blog post Capturing Logs in Docker Desktop describing how logging in Docker Desktop is implemented. What we can learn:

logwrite and memlogd are part of Linuxkit, and LinuxKit provides binaries of their packages as containers, with the logging infrastructure as linuxkit/memlogd. memlogd has a socket to query for logs at /run/guest-services/memlogdq.sock.

Be aware, that the latest-tag on linuxkit/memlogd is quite useless and pick a specific tag matching the running version in the VM; just try another version if the next step errors.

The solution

Run logread from linuxkit/memlogd and mount the VMs /run/guest-services/memlogdq.sock to the logreads expected default location at /var/run/memlogdq.sock. Tell logread to either show only new entries and follow, using -f, or dump all existing entries and follow with -F. Pipe this through something like grep, or a powershell equivalent and filter for kubelet:

# example assuming presence grep on windows
docker run --rm -it -v /run/guest-services/memlogdq.sock:/var/run/memlogdq.sock linuxkit/memlogd:014f86dce2ea4bb2ec13e92ae5c1e854bcefec40 /usr/bin/logread -F | grep -i kubelet

Tag 014f86dce2ea4bb2ec13e92ae5c1e854bcefec40 is working with Docker Desktop v4.11.1; just test which container version/tag works with your docker version.