I am getting following error while fetching metrics from prometheus, not able to find anything related on internet also not finding anything related in the logs .. please help what's going wrong here
Error updating options: "expanding series: block: 01G3T0PCNX96SX43TWVEGKTVG4: decode postings: invalid size"
i am running prometheus, version 2.35.0
I'm not sure whether this applies to your case, but for my future reference i'll quickly write what solved the issue in my case.
As stated before:
I have prometheus deployed as bitnami helm chart on my on-site k8s cluster. The error happens ever since i try to persist the data (because otherwise the charts disappear if the pod is restarted). We use CIFS as a storage system.
Further when i looked into the logs of the pod i found this
kubectl logs prometheus-prometheus-kube-prometheus-prometheus-0
ts=2022-09-09T11:00:35.345Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYH2WTM9X3TQD2T52RPEV6: invalid magic number 0"
ts=2022-09-09T11:00:54.184Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYHP0HMJZ1KVB608P3TV4E duration=2.838654027s
ts=2022-09-09T11:00:54.317Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYHP0HMJZ1KVB608P3TV4E: invalid magic number 0"
ts=2022-09-09T11:01:29.216Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYJR5DA60D17F3W2VAFZAT duration=2.899332494s
ts=2022-09-09T11:01:29.453Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYJR5DA60D17F3W2VAFZAT: invalid magic number 0"
ts=2022-09-09T11:02:32.354Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYMNTDYWWENPT2KSSNWN3H duration=2.900614585s
ts=2022-09-09T11:02:32.494Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYMNTDYWWENPT2KSSNWN3H: invalid magic number 0"
ts=2022-09-09T11:03:35.407Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYPKCES6PDQYB2MPCWFG44 duration=2.913101338s
ts=2022-09-09T11:03:35.554Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYPKCES6PDQYB2MPCWFG44: invalid magic number 0"
ts=2022-09-09T11:04:38.999Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYRGZ3TPP8Q03X7CKVSWHQ duration=3.444085613s
ts=2022-09-09T11:04:39.314Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYRGZ3TPP8Q03X7CKVSWHQ: read symbols: invalid checksum"
ts=2022-09-09T11:05:42.805Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYTF7KEBV6SPBYPR5SE8V3 duration=3.490813666s
ts=2022-09-09T11:05:42.992Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYTF7KEBV6SPBYPR5SE8V3: invalid magic number 0"
ts=2022-09-09T11:06:46.245Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYWDDHPSMCAZ4XK34DWH7D duration=3.252076412s
ts=2022-09-09T11:06:46.392Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYWDDHPSMCAZ4XK34DWH7D: invalid magic number 0"
ts=2022-09-09T11:07:49.278Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGYYBASWE0EC9DTPQD72H98 duration=2.884889018s
ts=2022-09-09T11:07:49.435Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGYYBASWE0EC9DTPQD72H98: invalid magic number 0"
ts=2022-09-09T11:08:52.155Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGZ08WWSMSG6PBZATX6HT1S duration=2.719327278s
ts=2022-09-09T11:08:52.269Z caller=db.go:829 level=error component=tsdb msg="compaction failed" err="compact head: reloadBlocks blocks: corrupted block 01GCGZ08WWSMSG6PBZATX6HT1S: invalid magic number 0"
ts=2022-09-09T11:09:55.542Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1662710400011 maxt=1662717600000 ulid=01GCGZ268E5K4S07EZS7GM9J16 duration=3.271842684s
ts=2022-09-09T11:09:55.893Z caller=head.go:844 level=info component=tsdb msg="Head GC completed" duration=223.829164ms
ts=2022-09-09T11:09:55.912Z caller=checkpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=0 to_segment=2 mint=1662717600000
ts=2022-09-09T11:10:03.499Z caller=head.go:1013 level=info component=tsdb msg="WAL checkpoint complete" first=0 last=2 duration=7.590667466s
Especially consider: compaction failed; corrupted block XXX invalid magic number 0
For this error case there are far more search results:
just to name a few. Especially the latter shows that this might well have to do with prometheus not supporting your specific filesystem, which is well possible with CIFS. From previous experience i knew that mounting options are crucial, so i added
mfsymlinks and cache=none to our storageClass. and indeed that resolved the issue.
btw. thanks @Machavity for keeping this thread "clean"
