How to change command-line-flags of prometheus in a docker container

Question

I followed this guide (https://www.jeffgeerling.com/blog/2021/monitor-your-internet-raspberry-pi) by Jeff Geerling to install an internet monitoring dashboard using prometheus and grafana running in docker containers.

Everything works great, but I noticed that the data is getting deleted after 15 days. After a quick search I found out that this is the default setting for the storage retention in prometheus.

I tried a lot by myself but I cannot find a way to change this setting.

Even though I found this tutorial (https://mkezz.wordpress.com/2017/11/13/prometheus-command-line-flags-in-docker-service/) which as far as I know should exactly tackle the problem I have but it doesn't work. I get the Error: Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. when running the first command mentioned.

Also I found this question (Increasing Prometheus storage retention) but I cannot use the best answer because my prometheus is running in the docker container.

Is there an easy way to set a command line flag for prometheus something like this --storage.tsdb.retention.time=30d?

This is the ReadMe-File I downloaded when I first installed it:

# Internet Monitoring Docker Stack with Prometheus + Grafana

> This repository is a fork from [maxandersen/internet-monitoring](https://github.com/maxandersen/internet-monitoring), tailored for use on a Raspberry Pi. It has only been tested on a Raspberry Pi 4 running Pi OS 64-bit beta.

Stand-up a Docker [Prometheus](http://prometheus.io/) stack containing Prometheus, Grafana with [blackbox-exporter](https://github.com/prometheus/blackbox_exporter), and [speedtest-exporter](https://github.com/MiguelNdeCarvalho/speedtest-exporter) to collect and graph home Internet reliability and throughput.

## Pre-requisites

Make sure Docker and [Docker Compose](https://docs.docker.com/compose/install/) are installed on your Docker host machine.

## Quick Start

`
git clone https://github.com/geerlingguy/internet-monitoring
cd internet-monitoring
docker-compose up -d
`

Go to [http://localhost:3030/d/o9mIe_Aik/internet-connection](http://localhost:3030/d/o9mIe_Aik/internet-connection) (change `localhost` to your docker host ip/name).



## Configuration

To change what hosts you ping you change the `targets` section in [/prometheus/pinghosts.yaml](./prometheus/pinghosts.yaml) file.

For speedtest the only relevant configuration is how often you want the check to happen. It is at 30 minutes by default which might be too much if you have limit on downloads. This is changed by editing `scrape_interval` under `speedtest` in [/prometheus/prometheus.yml](./prometheus/prometheus.yml).

Once configurations are done, run the following command:

    $ docker-compose up -d

That's it. docker-compose builds the entire Grafana and Prometheus stack automagically.

The Grafana Dashboard is now accessible via: `http://<Host IP Address>:3030` for example http://localhost:3030

username - admin
password - wonka (Password is stored in the `config.monitoring` env file)

The DataSource and Dashboard for Grafana are automatically provisioned.

If all works it should be available at http://localhost:3030/d/o9mIe_Aik/internet-connection - if no data shows up try change the timeduration to something smaller.

<center><img src="images/dashboard.png" width="4600" heighth="500"></center>



## Interesting urls

http://localhost:9090/targets shows status of monitored targets as seen from prometheus - in this case which hosts being pinged and speedtest. note: speedtest will take a while before it shows as UP as it takes about 30s to respond.

http://localhost:9090/graph?g0.expr=probe_http_status_code&g0.tab=1 shows prometheus value for `probe_http_status_code` for each host. You can edit/play with additional values. Useful to check everything is okey in prometheus (in case Grafana is not showing the data you expect).

http://localhost:9115 blackbox exporter endpoint. Lets you see what have failed/succeded.

http://localhost:9798/metrics speedtest exporter endpoint. Does take about 30 seconds to show its result as it runs an actual speedtest when requested.



## Thanks and a disclaimer

Thanks to @maxandersen for making the original project this fork is based on.

Thanks to @vegasbrianc work on making a [super easy docker](https://github.com/vegasbrianc/github-monitoring) stack for running prometheus and grafana.

This setup is not secured in any way, so please only use on non-public networks, or find a way to secure it on your own.

After further tinkering with it I found a docker-compose.yml file and I simply added under the command-section of prometheus the --storage.tsdb.retention.time=30d as shown here:

version: "3.1"

volumes:
    prometheus_data: {}
    grafana_data: {}

networks:
  front-tier:
  back-tier:

services:
  prometheus:
    image: prom/prometheus:v2.25.2
    restart: always
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
      - '--storage.tsdb.retention.time=30d'
    ports:
      - 9090:9090
    links:
      - ping:ping
      - speedtest:speedtest
    networks:
      - back-tier

  grafana:
    image: grafana/grafana
    restart: always
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning/:/etc/grafana/provisioning/
    depends_on:
      - prometheus
    ports:
      - 3030:3000
    env_file:
      - ./grafana/config.monitoring
    networks:
      - back-tier
      - front-tier

  ping:
    tty: true
    stdin_open: true
    expose:
      - 9115
    ports:
      - 9115:9115
    image: prom/blackbox-exporter
    restart: always
    volumes:
      - ./blackbox/config:/config
    command:
      - '--config.file=/config/blackbox.yml'
    networks:
      - back-tier

  speedtest:
    tty: true
    stdin_open: true
    expose:
      - 9798
    ports:
      - 9798:9798
    image: miguelndecarvalho/speedtest-exporter
    restart: always
    networks:
      - back-tier

  nodeexp:
    privileged: true
    image: prom/node-exporter
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    ports:
      - 9100:9100
    restart: always
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - --collector.filesystem.ignored-mount-points
      - "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)"
    networks:
      - back-tier

By then running docker-compose create followed by docker start internet-monitoring_prometheus_1 I can see under [Hostname of Server]:9090/status under Storage Retention 30 days. Is that the way it should be done, as I think I found my solution.