how to fix ceph warning "storage filling up"

Question

i have a cluster ceph and in monitoring tab dashboard show me warning "storage filling up"

alertname
storage filling up
description
Mountpoint /rootfs/run on ceph2-node-03.fns will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.

but all devices is free

[root@ceph2-node-01 ~]# ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA    OMAP    META     AVAIL    %USE  VAR   PGS  STATUS
 0    hdd  0.01900   1.00000   20 GiB   61 MiB  15 MiB     0 B   44 MiB   20 GiB  0.30  0.92    0      up
 3    ssd  0.01900   1.00000   20 GiB   69 MiB  15 MiB   5 KiB   53 MiB   20 GiB  0.33  1.04    1      up
 1    hdd  0.01900   1.00000   20 GiB   76 MiB  16 MiB   6 KiB   60 MiB   20 GiB  0.37  1.15    0      up
 4    ssd  0.01900   1.00000   20 GiB   68 MiB  15 MiB   3 KiB   52 MiB   20 GiB  0.33  1.03    1      up
 2    hdd  0.01900   1.00000   20 GiB   66 MiB  16 MiB   6 KiB   50 MiB   20 GiB  0.32  1.00    0      up
 5    ssd  0.01900   1.00000   20 GiB   57 MiB  15 MiB   5 KiB   41 MiB   20 GiB  0.28  0.86    1      up
                       TOTAL  120 GiB  396 MiB  92 MiB  28 KiB  300 MiB  120 GiB  0.32
MIN/MAX VAR: 0.86/1.15  STDDEV: 0.03

what should i do to fix this warning? this is bug or ...?

may be I can try to help. First please tell me are you using cephadm/containers or distro packages? — BZKN, Feb 02 '22 at 06:24
The predictions from prometheus are not really helpful, to be honest. Did you write a significant amount of data within a short period of time into the cluster? If so, you could wait it out until the alert clears (probably). The ruleset looks something like this: `predict_linear(node_filesystem_free_bytes[2d], 3600 * 24 * 5) * on(instance) group_left(nodename) node_uname_info < 0`. It tries to predict the filling rate based on the last two days. — eblock, Feb 02 '22 at 10:23

how to fix ceph warning "storage filling up"

0 Answers0