6

"12h" old logs are not deleting even after specifying "retention_period: 12h" due to this I'm facing storage issue because, log are not deleting. please help me with config

loki:
  enabled: true
  isDefault: true
  table_manager:
    retention_deletes_enabled: true
    retention_period: 12h
Vinayak Pawar
  • 93
  • 1
  • 4
  • 1
    What exactly do you mean by: "12h" old logs are not deleting? I am asking because Loki is not supposed to delete any log file. To delete those files, you would use some kind of log rotation provided by your logging framework. (BTW: Getting those files is part of the Promtail configuration not part of Loki.) – Sascha Doerdelmann Jul 27 '22 at 13:46
  • 2
    Actually, My goal is to deleted logs which are older than 12h, it should be deleted from grafana dashboard as well as from server storage because I m facing storage issues as my log are not deleting, and occupying huge space in server storage is getting full – Vinayak Pawar Jul 29 '22 at 05:14

3 Answers3

7

I have the following configurations related to log retention:

...
compactor:
  ...
  retention_enabled: true
...
limits_config:
  ...
  retention_period: 90d
...

See more about Loki configuration in the documentation here

Marcelo Ávila de Oliveira
  • 19,950
  • 3
  • 39
  • 50
3

Grafana Loki Storage Retention

By default, when table_manager.retention_deletes_enabled or compactor.retention_enabled flags are not set, then logs sent to Loki live forever.

The minimum retention period is 24h.

The retention period must be a multiple of the index and chunks table period, configured in the period_config block.

Sample config:

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

table_manager:
  retention_deletes_enabled: true
  retention_period: 24h

The other way to setup the retention in Grafana Loki is via the Compactor.

limits_config:
  retention_period: 168h

compactor:
  working_directory: /data/retention
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

Note that retention is only available if the index period is 24h.

To purge data immediately you could set the retention_delete_delay interval to 1m. This is a delay after which chunks will be fully deleted during retention.

Do not forget to restart Loki after changing the configuration:

systemctl restart loki
systemctl status loki
Vladislav
  • 1,696
  • 27
  • 37
  • `This is a delay after which chunks will be fully deleted during retention` I do not understand. So the logs will be deleted after retention_delete_delay or after limits_config: retention_period? Why are there two settings? – KamilCuk Aug 16 '23 at 09:32
  • 2
    `retention_delete_delay` just delays deleting defined in `limits_config\retention_period`. The reason for that is described in https://grafana.com/docs/loki/latest/operations/storage/retention/ as: "1. ... deleting chunks instantly could lead to components still having reference to old chunks and so they could fails to execute queries. Having a delay allows for components to refresh their store and so remove gracefully their reference of those chunks. 2. It provides a short window of time in which to cancel chunk deletion in the case of a configuration mistake." – Vladislav Aug 17 '23 at 12:23
0

From the Grafana storage documentation:

With the exception of the filesystem chunk store, Loki will not delete old chunk stores. This is generally handled instead by configuring TTLs (time to live) in the chunk store of your choice (bucket lifecycles in S3/GCS, and TTLs in Cassandra). Neither will Loki currently delete old data when your local disk fills when using the filesystem chunk store – deletion is only determined by retention duration.

This quote should answer your question. You need to configure bucket lifecycles in your storage provider to be able to delete chunk stores.

Ri1a
  • 737
  • 9
  • 26