2

I was curious if it is possible to move completely off hostpaths and use local volumes (persistent volumes) instead. However, there doesn't seem to be a way to include volumeClaimTemplates into a daemonset.

Statefulsets provide volumeClaimTemplates, but they require the replicas to be manually defined, as opposed to auto scaling to as many nodes as can be found.

Is there a way to automatically scale to all nodes, and also create a pvc per replica?

Related question where they decided to use statefulsets and give up autoscaling: Handling PersistentVolumeClaim in DaemonSet

kittydoor
  • 101
  • 12
  • What are you trying to achieve? StatefulSet is made for stateful workload. Kubernetes abstract away the underlying infrastructure, so you should not need to think so specific about each node. – Jonas Dec 27 '19 at 18:26
  • @Jonas I'm trying to scale a distributed storage solution (in this case glusterfs) across all nodes that have storage disks intended to be used by this solution. For this purpose, I wish to manually create local volumes on these nodes and label them as distributed storage volumes, and then have the storage solution automatically scale across all available nodes with matching volumes. – kittydoor Dec 27 '19 at 19:22
  • HostPaths is directories on the node, https://kubernetes.io/docs/concepts/storage/volumes/#hostpath and glusterfs setup already has a DaemonSet to use it https://github.com/gluster/gluster-kubernetes ..so I understand that you want to do some modifications.... but volumeClaims is a higher abstraction, so I think it should be avoided when you want to provide another storage system? – Jonas Dec 27 '19 at 19:37
  • @Jonas The point of local volumes is to keep the k8s manifest detached from infrastructure details. It let's me separate the request for local space and where exactly on disk that is. So, portability and visibility. – kittydoor Dec 28 '19 at 14:26
  • Yes, exactly - that is for application storage. GlusterFS is a storage system itself - so this is a different use case, exactly therefor it is deployed as a `DaemonSet` with more tightly storage coupling. – Jonas Dec 28 '19 at 14:43
  • @Jonas Whether it is a good idea for something like GlusterFS to be deployed as a daemonset with hostpaths, or to use local volumes is a separate discussion. Surely there is value in having that discussion, and I see your argument, but it doesn't answer my question above which is detached from any specific application such as GlusterFS. – kittydoor Dec 29 '19 at 04:49
  • StatefulSet can not distribute pods like a DaemonSet - because it does not make sense. There is not volumeClaimTemplate in DaemonSet - because it does not make much sense. The combination you are trying to achieve is strange. This is totally relevant to your question. – Jonas Dec 29 '19 at 09:31
  • @KatherineDoor please take a look at: [Kubernetes: local persistent volumes](https://kubernetes.io/blog/2019/04/04/kubernetes-1.14-local-persistent-volumes-ga/). – Dawid Kruk Jan 02 '20 at 08:26
  • @Jonas you are overly focusing on my choice of mentioning GlusterFS as an example here. There are cases like netdata, or custom services for remote management, monitoring/exposing data, etc that also benefit from this. The intent of separating a request for persistent storage, and where it exists (in the local machine, or in any remote system) or the details behind it (replicated, single source of truth), and enabling this to be automatically managed via storage classes and provisioners is very useful to me, for statefulsets and daemonsets both. – kittydoor Jan 04 '20 at 04:15
  • @DawidKruk yes, that is what I wish to utilize. The problem is that, volumeClaimTemplates which are required to bind to local volumes which are a type of PV (as in, templating so that all replicas/pods do not share the same volume, and instead get individual volumes) is a statefulset exclusive feature at the moment. – kittydoor Jan 04 '20 at 04:16
  • ReplicaSet and StatefulSet-pods is scheduled to nodes that has available PersistentVolumes... this can not be done with DaemonSet since scheduling is forced different – Jonas Jan 04 '20 at 10:04
  • It would not be a good situation if pods from a DaemonSet can not be scheduled because it is blocked by unavailable PersistentVolumes. When using DaemonSet you expect scheduling following NodeSelector. – Jonas Jan 04 '20 at 10:17
  • @Jonas when using non local and dynamically provisioned persistent volumes (e.g. Rook+Ceph), this is not an issue. If a replica can not be scheduled onto a node due to a lack of available persistent volumes, this is something that should be alerted on and fixed in the storage infra. – kittydoor Jan 04 '20 at 18:14

1 Answers1

3

There is no possibility for now to use PersistentVolumes in Daemon Set.

There was a feature request for it on Github but unfortunately it was closed. Link to this request: volumeClaimTemplates available for Daemon Sets

There is a comment in the link above which is describing this topic a bit more.

Dawid Kruk
  • 8,982
  • 2
  • 22
  • 45
  • Hmm, other than extending kubernetes with an additional controller, it seems this is not possible to do directly at the moment. Are you aware of any sigs or 3rd party projects on this? Or any solutions other than hostPaths anyone has decided to implement? – kittydoor Jan 04 '20 at 04:26
  • I am not aware of any other 3rd party tools that can accomplish what you intend to do. – Dawid Kruk Jan 06 '20 at 12:16