Decrypting Docker Container Names in Grafana with Loki and Promtail

Question

My task is to configure Docker log monitoring for around 70 containers. Consolidating all logs into a single panel is not ideal, especially during stress tests and monitoring Docker applications. I want to be able to select a Docker container application in a panel and see the corresponding logs. I have achieved this, but the problem is that the container names are encrypted, and I need to decrypt them.

My question is: How can I decrypt the container names using the Loki data source and Promtail?

Additionally, I noticed that the Prometheus data source displays decrypted container names, but the Loki data source does not. Here are my configurations for both:

Also, I am running Loki and Promtail like normal application with configured systemd daemon. Loki:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096


ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
  - from: 2021-03-08
    store: boltdb
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h

storage_config:
  boltdb:
    directory: /tmp/loki/index

Promtail:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: cadvisor
  static_configs:
  - targets: ['localhost:8080/containers/']
    labels:
      job: cadvisonr
      __path__: /containers
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        output: log
        stream: stream
        attrs:
  - json:
      expressions:
        tag:
      source: attrs
  - regex:
      expression: (?P<container_name>(?:[^|]*[^|]))
      source: tag
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      # tag:
      stream:
      container_name:
  - output:
      source: output

Also, I know about loki-driver-docker plugin, but I tried to run it and nothing changes for me, and documentation is terrible, it says just run it and almost nothing about configuration or how to check if it is working correct.

So, I would be glad to hear any ideas and suggestions. Forum is the last chance for me to solve this problem.

UPD: Currently, it looks like this:

And I want it to be decrypted like in cadvisor:

`__path__: /var/lib/docker/containers/` But why are you not using docker config in promtail? — KamilCuk, Apr 03 '23 at 12:32
What do you mean? I have this line for collecting logs: ```__path__: /var/lib/docker/containers/*/*log``` — worrum, Apr 03 '23 at 12:36
https://grafana.com/docs/loki/latest/clients/promtail/configuration/#docker_sd_config You are reading _files_, you will not get any info from files, files are what they are - it's just a SHA of container. — KamilCuk, Apr 03 '23 at 12:41
So, promtail which was build and is running from source, will not be able to collect logs from docker container because can't use this parameter ```--log-driver json-file --log-opt max-size=10m```? And, in the config file I should use - ```__meta_docker_container_name``` parameter to get docker container name? How to select specified logs files in Grafana then? Sorry, I am Junior and don't have much experience, so I don't understand you — worrum, Apr 03 '23 at 12:47

score 5 · Accepted Answer · answered Apr 03 '23 at 12:51

It works like this:

promtail has backends <something>_configs
every backend is slightly different
loki collects lines of logs
every line of logs in loki has "labels"
promtail backends can convert some internal info to "labels" and send that loki
promtail file backend only sees files, so static_config only sees files
promtail docker backend connects to docker deamon and actually knows something about those dockers

Promtail config may look like this:

  - job_name: "docker"
    docker_sd_configs:
      - host: "unix:///var/run/docker.sock"
        refresh_interval: "1s"
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        target_label: "container_name"
      - source_labels: ['__meta_docker_container_id']
        target_label: "container_id"
      # etc.

Then in grafana-loki you can filter the logs using LogQL language, like:

logcli '{container_name="something"}'

can you help me again? I will close topic after this. Your solution is perfect, it works on my localhost, but I have two machines and each one has it's own containers. I am using one promtail instance at one machine and scrape metrics/logs from another directly specifying it's ips - 10.129.11.20 and 10.129.11.22. How to implement this solution to docker.sock, because as documentation said - ```Use unix:///var/run/docker.sock for a local setup```. — worrum, Apr 03 '23 at 14:08
https://stackoverflow.com/questions/62757127/what-are-the-possible-formats-of-the-docker-host-urls — KamilCuk, Apr 03 '23 at 14:11

Decrypting Docker Container Names in Grafana with Loki and Promtail

1 Answers1