I try to use Grafana Agent (version 0.35) instead of Promtail to send logs to Grafana Loki (version 2.8.4). I want to send Docker container's logs, so I configured docker_sd_configs. But after starting Grafana Agent I couldn't find any logs with Grafana. But I can see an error message in Grafana Agent's logs.
Aug 25 10:02:15 test grafana-agent[12795]: ts=2023-08-25T10:02:15.806026191Z caller=client.go:430 level=error component=logs logs_config=default component=client host=172.31.11.242:3100 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): entry for stream '{container=\"keycloak\", host=\"test\", source=\"stdout\"}' has timestamp too old: 2023-08-16T16:20:32Z, oldest acceptable timestamp is:2023-08-18T10:02:15Z"
Configuration
logs:
configs:
- name: default
positions:
filename: /tmp/positions.yaml
clients:
- url: http://172.31.11.242:3100/loki/api/v1/push
external_labels:
host: test
scrape_configs:
- job_name: docker
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 5s
relabel_configs:
- source_labels: ['__meta_docker_container_name']
regex: '/(.*)'
target_label: 'container'
- source_labels: [__meta_docker_container_log_stream]
target_label: source
Research
I didn't have the problem with pure Promtail version 2.8.4. At least I didn't notice it.
I read Loki/Promtail : parsing timestamp that are too old, but I don't want to change Loki's configuration to allow old logs. I just want to skip all too old logs.
After restarting the Docker containers, Grafana Agent is reading all new logs. But I don't want to restart every container after installing Grafana Agent.
I read journal reading: Timestamp too old errors, but this fix is only for the
journalconfiguration.I read Timestamp error, but it is not about Docker container logs. Also I didn't find any official documentation about
eventTime.
Question
How to configure Grafana Agent to only skip too old logs?
