How to run Dgraph on bare-metal Kubernetes cluster

Question

I am trying to setup Dgraph in HA Cluster but it won't deploy if no volumes are present.

When directly applying the provided config on a bare-metal cluster won't work.

$ kubectl get pod --namespace dgraph
dgraph-alpha-0                      0/1     Pending     0          112s
dgraph-ratel-7459974489-ggnql       1/1     Running     0          112s
dgraph-zero-0                       0/1     Pending     0          112s


$ kubectl describe pod/dgraph-alpha-0 --namespace dgraph
Events:
  Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
  Warning  FailedScheduling  <unknown>  default-scheduler  error while running "VolumeBinding" filter plugin for pod "dgraph-alpha-0": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling  <unknown>  default-scheduler  error while running "VolumeBinding" filter plugin for pod "dgraph-alpha-0": pod has unbound immediate PersistentVolumeClaims

Anyone else has this problem? I've been experiencing this issue for several days now and can not find a way around this. How can I have Dgraph use cluster's local storage?

Thanks

I don't see a definition of a PersistentVolume in the yaml you linked above, this needs to exist in order to fulfill the claim — omricoco, May 19 '20 at 19:58
the yaml uses the storage class to create the PersistentVolume you can test the yaml out on a cluster that has storageclass setup and you will see the pv created..it is a problem here because am on bare metall k8s cluster and do not have a way to make storageclass work — uberrebu, May 19 '20 at 20:58
If you have a storage class, this will solve your issue: https://stackoverflow.com/a/52669115/1970639 — omricoco, May 20 '20 at 07:18

score 1 · Accepted Answer · edited Oct 06 '20 at 05:29

Found a working solution myself.

I have to manually create the pv and pvc, then Dgraph can use them during deployment.

Here is the config I used to create the needed storageclass, pv and pvc

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-dgraph-dgraph-alpha-0
  labels:
    type: local
spec:
  storageClassName: local
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/dgraph/alpha-0"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-dgraph-dgraph-alpha-1
  labels:
    type: local
spec:
  storageClassName: local
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/dgraph/alpha-1"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-dgraph-dgraph-alpha-2
  labels:
    type: local
spec:
  storageClassName: local
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/dgraph/alpha-2"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-dgraph-dgraph-zero-0
  labels:
    type: local
spec:
  storageClassName: local
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/dgraph/zero-0"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-dgraph-dgraph-zero-1
  labels:
    type: local
spec:
  storageClassName: local
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/dgraph/zero-1"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-dgraph-dgraph-zero-2
  labels:
    type: local
spec:
  storageClassName: local
  capacity:
    storage: 8Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/dgraph/zero-2"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datadir-dgraph-dgraph-alpha-0
spec:
  storageClassName: local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datadir-dgraph-dgraph-alpha-1
spec:
  storageClassName: local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datadir-dgraph-dgraph-alpha-2
spec:
  storageClassName: local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datadir-dgraph-dgraph-zero-0
spec:
  storageClassName: local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datadir-dgraph-dgraph-zero-1
spec:
  storageClassName: local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datadir-dgraph-dgraph-zero-2
spec:
  storageClassName: local
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi

When Dgraph is deployed it latches at the pvc

$ kubectl get pvc -n dgraph -o wide
NAME                            STATUS   VOLUME                          CAPACITY   ACCESS MODES   STORAGECLASS   AGE     VOLUMEMODE
datadir-dgraph-dgraph-alpha-0   Bound    datadir-dgraph-dgraph-zero-2    8Gi        RWO            local          6h40m   Filesystem
datadir-dgraph-dgraph-alpha-1   Bound    datadir-dgraph-dgraph-alpha-0   8Gi        RWO            local          6h40m   Filesystem
datadir-dgraph-dgraph-alpha-2   Bound    datadir-dgraph-dgraph-zero-0    8Gi        RWO            local          6h40m   Filesystem
datadir-dgraph-dgraph-zero-0    Bound    datadir-dgraph-dgraph-alpha-1   8Gi        RWO            local          6h40m   Filesystem
datadir-dgraph-dgraph-zero-1    Bound    datadir-dgraph-dgraph-alpha-2   8Gi        RWO            local          6h40m   Filesystem
datadir-dgraph-dgraph-zero-2    Bound    datadir-dgraph-dgraph-zero-1    8Gi        RWO            local          6h40m   Filesystem

bh90210 · Answer 2 · 2020-10-07T01:57:29.550

Dgraph's configs assume a Kubernetes cluster with a working volume plugin (provisioner). In managed Kubernetes offerings (aws, GKE, DO etc) this step is already taken care by the provider.

I think the goal should be to achieve par functionality with cloud providers, that is the provisioning must be dynamic (in contrast for example with OP's own answer which is correct but statically provisioned - k8s docs).

When running bare-metal you have to manually configure a volume plugin before being able to dynamically provision volumes (k8s docs) and thus use StatefulSets, PersistentVolumeClaims etc. Thankfully there are many provisioners available (k8s docs). For out of the box support for dynamic provisioning every item in the list that has 'Internal Provisioner' checked will do.

So while the problem has many solutions I ended up using NFS. To achieve dynamic provisioning I had to use an external provisioner. Hopefully this is as simple as installing a Helm Chart.

Install NFS (original guide) on master node.

ssh via terminal and run

sudo apt update
sudo apt install nfs-kernel-server nfs-common

Create the directory Kubernetes is going to use and change ownership

sudo mkdir /var/nfs/kubernetes -p
sudo chown nobody:nogroup /var/nfs/kubernetes

Configure NFS

Open the file /etc/exports

sudo nano /etc/exports

Add the following line at the bottom

/var/nfs/kubernetes  client_ip(rw,sync,no_subtree_check)

Replace client_ip with you master node ip. In my case this IP was the DHCP server IP leased by my router to the machine running master node (192.168.1.7)

Restart NFS to apply the changes.

sudo systemctl restart nfs-kernel-server

After setting up NFS on master and assuming Helm is present, installing the provisioner is as simple as running

helm install  nfs-provisioner --set nfs.server=XXX.XXX.XXX.XXX --set nfs.path=/var/nfs/kubernetes --set storageClass.defaultClass=true stable/nfs-client-provisioner

Replace nfs.server flag with the appropriate IP/hostname of master node/NFS server.

Note flag storageClass.defaultClass has to be true in order for Kubernetes to default use the plugin (provisioner) for volume creation.

Flag nfs.path is the same path as the one created in step 2.

In case Helm complains that can not find the chart run helm repo add stable https://kubernetes-charts.storage.googleapis.com/

After successfully completing previous steps proceed installing Dgraph configs as described in their docs and enjoy you bare-metal dynamically provisioned cluster with out-of-the-box working Dgraph deployment.

Single server

kubectl create --filename https://raw.githubusercontent.com/dgraph-io/dgraph/master/contrib/config/kubernetes/dgraph-single/dgraph-single.yaml

HA Cluster

kubectl create --filename https://raw.githubusercontent.com/dgraph-io/dgraph/master/contrib/config/kubernetes/dgraph-ha/dgraph-ha.yaml

How to run Dgraph on bare-metal Kubernetes cluster

2 Answers2