2

As per prometheus storage.md , the recommendation is not to use nfs storage as persistent volume for prometheus.

But solutions like prometheus operator and openshift shows examples which uses nfs as persistent volumes for prometheus.

So what am I missing here? If nfs is not recommended then why do these tools share examples to use nfs as the storage options for prometheus?

Does anyone know what could be the nfs alternative for NetApp/ Trident for prometheus?

swetad90
  • 784
  • 1
  • 13
  • 34

3 Answers3

4

The example in the prom-operator docs is just a hypothetical to show how to manually control the storage provisioning. NFS is generally an option of last resort in all cases :) Check out https://kubernetes.io/docs/concepts/storage/persistent-volumes/ for more general information on how to use each of the various PV plugins (or if none of those work, look up CSI stuffs), but for NetApp you would probably use the iSCSI interface.

coderanger
  • 52,400
  • 4
  • 52
  • 75
  • I am working on setting up NFS mount for Prometheus-Grafana operator and came across this post while researching various approaches since default options are storing data in the worker nodes, causing it to fill up quickly. We have on-prem clusters, so wondering why is it a bad idea to use NFS mounts for PV set up, are there other options? – cnu Nov 02 '20 at 21:33
  • 1
    NFS is a relatively inefficient network storage protocol. It’s main benefit is broad standardization because it’s so old, and multi-writer support. iSCSI or other newer network block protocols better answer the first and the second is not wanted for Prometheus so it provides no benefit. – coderanger Nov 03 '20 at 04:29
  • Got it, will look into iSCSI option, or go with hostPath for now to use worker nodes storage. – cnu Nov 03 '20 at 13:40
1

I have been using prometheus with NFS (NetApp) mounted disk on ubuntu for 4+ years already, 24/7 operation, with rather high frequency monitoring and relatively large dataset (currently at 500GB, but was scraped couple of times before).

Without single glitch, neither storage related, nor otherwise.

Also, from my experience, properly set NFS mount, from reliable storage server (NetApp/Ontap), with solid networking, is 'as good as local disk' for all of your favorite data hoarders. In my case, mysql, grafana, and prometheus all sit happily on ubuntu nfs mount... At Ontap side, the snapshots and optionally snapmirrors will cover most if not all backup needs.

Again, this is from my personal experience...

R. Simac
  • 737
  • 9
  • 9
  • Lucky you! ... I wish I were that lucky. I'm rolling out prometheus on Azure as container app and Azure File Shares. Corruption happens like.. monthly. I'm looking for alternatives. – Lam Le Sep 16 '21 at 06:19
0

If you need to use NFS storage for Prometheus-like system, then give a try to VictoriaMetrics. It works in production on NFS and Amazon EFS without issues.

valyala
  • 11,669
  • 1
  • 59
  • 62