1

I have a Kubernetes cluster with three control and three worker nodes. To install Ceph I'm using rook. After the installation, only the worker nodes are included in the Ceph cluster - but not the controller nodes.

The configuration snippet:

storage:                                                                                                                                                                       
  useAllNodes: true
  useAllDevices: true

Is there an easy way to also include the controller nodes in the Ceph cluster? Or: do I need to explicitly list all controller and worker nodes in the storage nodes list?

Andreas Florath
  • 4,418
  • 22
  • 32
  • By default in kubernetes cluster pods are not scheduled on the master node, you need to remove the taint from the master to start scheduling pods on master. its better not to use master node to run apps – Hackaholic Nov 28 '21 at 13:59
  • Thanks a lot for your hint! This pointed me in the right direction. Nevertheless completely removing the taints was not necessary - see my answer. – Andreas Florath Dec 02 '21 at 10:18
  • you you can add toleration to the pods – Hackaholic Dec 03 '21 at 18:30

1 Answers1

1

The root cause for the problem was, that the master nodes were tainted in a way that no 'normal' pod was allowed to run there.

Adapting the cluster.yaml configuration of rook and adding the following lines which explicitly allowed the ceph pods run also on master nodes solved the problem:

placement:
  all:
    tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master

Thanks a lot to Hackaholic who pointed me in this direction.

Update 2024-03: The above solution stopped working for me using kubernetes 1.26.3 and kubespray 1.10.13. IMHO one of the reasons is, that the master label is deprecated. Using the deprecated master and new control-plane label works for me - specifying only one does not. So the current solution looks like:

placement:                                                                                                                                       
  all:                                                                                                                                           
    tolerations:                                                                                                                                 
      - effect: NoSchedule                                                                                                                       
        key: node-role.kubernetes.io/control-plane                                                                                               
      - effect: NoSchedule                                                                                                                       
        key: node-role.kubernetes.io/master
Andreas Florath
  • 4,418
  • 22
  • 32