Why is Loki's Docker Driver Client stopping to log after some time?

Question

I want to send logs of my Docker containers to Grafana Loki. Therefore, I installed Loki's Docker Driver Client and started my containers with it. First I can see logs, but after some time I see no more logs.

Installation

I installed Loki's Docker Driver Client as a Docker plugin on my Docker Engine (version 20.10.2):

$ docker plugin install grafana/loki-docker-driver:master-54d1d3b --alias loki --grant-all-permissions

I didn't use the tag lastest, because of the bug Unable to connect to logging plugin in Swarm

Configuration

I started my Docker containers with Loki's Docker Driver Client as log driver:

$ docker container run
  --log-driver=loki
  --log-opt loki-url="$LOKI_URL"
  --log-opt loki-retries=5
  --log-opt loki-batch-size=400
  --log-opt max-size="10m"
  --log-opt max-file=5
  --detach
  --name $CONTAINER_NAME
  --restart unless-stopped
  $IMAGE:$TAG

I also added json-log driver's max-size and max-file to limit disk space, see Configuring the Docker Driver.

Problem

First I could see logs in Grafana and in command line with docker container logs, but after some time no more logs were shown. If I tried to look into the logs on Docker host and I saw an error:

$ docker container logs 75d4b13eb3e8
error from daemon in stream: Error grabbing logs: error getting log reader: LogDriver.ReadLogs: logger does not exist for 75d4b13eb3e8203b9247ecdeb41fdf495cc8fea7dcfc4775fd8261263b1dcd32

Research

I looked into the directories of the containers (see Where is a log file with logs from a container?), but I couldn't see any log files:

$ sudo ls /var/lib/docker/containers/75d4b13eb3e8203b9247ecdeb41fdf495cc8fea7dcfc4775fd8261263b1dcd32
checkpoints  config.v2.json  hostconfig.json  hostname  hosts  mounts  resolv.conf  resolv.conf.hash

I also checked the log path (see Get an instance’s log path), but it was empty:

$ docker inspect --format='{{.LogPath}}' 75d4b13eb3e8

I found container's logs in plugin's directory (see Loki log driver not storing logs as files on disk, even with keep-file: true), but the log files don't change anymore:

$ sudo ls -la /var/lib/docker/plugins/eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288/rootfs/var/log/docker/75d4b13eb3e8203b9247ecdeb41fdf495cc8fea7dcfc4775fd8261263b1dcd32
total 912
drwxr-xr-x  2 root root   4096 Jan 22 12:59 .
drwxr-xr-x 17 root root   4096 Jan 22 15:46 ..
-rw-r-----  1 root root 923177 Jan 22 13:34 json.log

I looked into Docker daemon's logs (see Read the logs) and found errors and a warning (at the same time logging stopped):

$ sudo journalctl -u docker.service | grep eac33cc9913c
[...]
[...]level=error msg="panic: send on closed channel" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="goroutine 153 [running]:" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="main.(*loki).Log(0xc0000c5e00, 0xc0001d81c0, 0xc0000c5e80, 0x0)" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="\t/src/loki/cmd/docker-driver/loki.go:69 +0x2fb" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="main.consumeLog(0xc0002c0480)" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="\t/src/loki/cmd/docker-driver/driver.go:165 +0x4c2" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="created by main.(*driver).StartLogging" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="\t/src/loki/cmd/docker-driver/driver.go:116 +0xa75" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=warning msg="Unable to connect to plugin: /run/docker/plugins/eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288/loki.sock/LogDriver.StopLogging: Post http://%2Frun%2Fdocker%2Fplugins%2Feac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288%2Floki.sock/LogDriver.StopLogging: EOF, retrying in 1s"
[...]

What did I do wrong?

Answer 1

I was experiencing the same issue.

My only differences in configuration are that I'm trialing the latest Enterprise Edition (19.03) as it brings dual logging capability although this is also supported in the latest CE versions, and I'm using the latest Loki Docker driver client now that the Github issue previously mentioned has been resolved.

I ended up setting the log-opts properties no-file and keep-file in docker-compose.yml:

logging:
  driver: "loki"
  options:
    loki-url: "http://${LOKI_URL}:3100/loki/api/v1/push"
    loki-batch-size: "400"
    no-file: "false"
    keep-file: "true"
    max-size: "5m"
    max-file: "3"

Since making this change I am receiving logs in Loki and can still use docker container logs and docker service logs on my Docker hosts.

no-file: "false" tells the driver to continue creating logs on disk and keep-file: "true" tells the driver to keep json logs if the container is stopped (by default files are removed).

Note: Originally I was adding these settings to /etc/docker/daemon.json on the host but would still see the error getting log reader issue, I had to switch to specifying the log driver per container/swarm service.

Answer 2

Regarding this issue

First I could see logs in Grafana and in command line with docker container logs, but after some time no more logs were shown.

On Grafana please select Query type: Range not Instant and you will see all the logs for the selected period of time, if exists in loki.