46

I have Prometheus server installed on my AWS instance but the data is been removed automatically after 15 days. I need to have data for a year or months. Is there anything I need to change in my prometheus configuration?

Or do I need any extensions like Thanos? I am new to Prometheus so please be easy on the answers.

codeforester
  • 39,467
  • 16
  • 112
  • 140
Rohit Bharati
  • 563
  • 1
  • 4
  • 5
  • 3
    Prometheus supports remote_write mechanism for replicating data to [long-term storage systems](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage), so the data could be queried later from these systems. The most promising systems are Cortex, m3db and [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/). – valyala Dec 05 '20 at 21:44

5 Answers5

44
  1. Edit the prometheus.service file

vi /etc/systemd/system/prometheus.service

  1. add "--storage.tsdb.retention.time=1y" below to "ExecStart=/usr/local/bin/prometheus \" line.

So the config will look like bellow for 1 year of data retention.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.external-url=http://34.89.26.156:9090 \
    --storage.tsdb.retention.time=1y
[Install]
WantedBy=multi-user.target
Noam Yizraeli
  • 4,446
  • 18
  • 35
Prabhu
  • 666
  • 6
  • 11
  • Is there a downside to increasing the retention time, slowness, or something else? – BabyDuck May 12 '23 at 09:13
  • Depends on your data volume, if the data size will be huge, you will need more ram/cpu to fetch the same in to metrics – Prabhu Jul 25 '23 at 06:45
31

There's the --storage.tsdb.retention.time flag that you can set when you start Prometheus. It defines how long data is kept in the time-series database (TSDB). The default is 15 days.

So, to increase the retention time to a year, you should be able to set this to something like:

--storage.tsdb.retention.time=1y
# or
--storage.tsdb.retention.time=365d

See the Prometheus documentation.

Chin
  • 19,717
  • 37
  • 107
  • 164
weibeld
  • 13,643
  • 2
  • 36
  • 50
  • 2
    Can this be set in the yml? – tmm1 Mar 17 '20 at 22:10
  • No, it can only be set with command-line flags. It's an [immutable system parameter](https://prometheus.io/docs/prometheus/latest/configuration/configuration/). – weibeld Mar 20 '20 at 17:47
  • 12
    Be careful with this answer, 'm' stands for 'minutes' not 'months' as described here: https://manpages.debian.org/unstable/prometheus/prometheus.1.en.html – vitaminwater May 13 '20 at 10:36
  • 2
    To add some context to this answer, there's a reason the default is 15 days. Per the [docs](https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects), _Prometheus's local storage is not meant as durable long-term storage._ Ensure your storage is suitably configured with durability if one year's worth of metrics is essential to operations. – Witt Aug 26 '20 at 23:28
  • @vitaminwater from the docs m seems to be minutes and there's no way to specify months (but you can specify weeks by 'w') – rkachach Sep 07 '22 at 11:41
  • From [cmd/prometheus/main.go](https://github.com/prometheus/prometheus/blob/bef6556ca566b47be52088466f36c1eaab4a2e45/cmd/prometheus/main.go) on GitHub: `Units Supported: y, w, d, h, m, s, ms.`. – weibeld Sep 08 '22 at 12:15
9

adding below in deployment yml file allowed me change storage retention days

image: 'your/image path' 
args:
  - '--storage.tsdb.path=/prometheus'
  - '--storage.tsdb.retention.time=45d'
  - '--config.file=/etc/prometheus/prometheus.yml'
codeforester
  • 39,467
  • 16
  • 112
  • 140
PrashanthK
  • 99
  • 1
  • 1
7

On Debian you do not have to edit the systemd-config. You simply can add the arguments to

/etc/default/prometheus

like so:

# Set the command-line arguments to pass to the server.
ARGS="--storage.tsdb.retention.time=60d"
rbs
  • 1,087
  • 2
  • 17
  • 24
  • 1
    in my case is different, I download the Prometheus from Bitnami using helm. and upgrade the values by editing the values to retention up to 10d, because the default is 10d. but seems not working at all. it does not even survive in 24 hours. i use ubuntu and install my Prometheus inside of the cluster. i tried to find in /etc/default/prometheus to add the arg. Unfortunately, I do not have the Prometheus file there. any solution? import the Prometheus or snapshot Grafana, the only temporary solution that I ever had, but still not tried yet. – newcomers Mar 11 '22 at 00:39
0

On Ubuntu with prometheus installed via snap, edit /var/snap/prometheus/current/daemon_arguments and add --storage.tsdb.retention.time=1y to the ARGS= line at the top.

Restart the prometheus snap: systemctl restart snap.prometheus.prometheus.service

chouse
  • 11
  • 1