How to identify the storage space left in a PVC?

Question

I have a pod with PVC request of 10Gi and is successfully bound to PV(both definition below)

I came across the accepted answer of similar question, which suggests to run kubectl -n <namespace> exec <pod-name> df Upon doing the same i got the following -

pavan@p1: kubectl exec mysql-deployment-95f7dd544-mmjv9 df
Filesystem     1K-blocks    Used Available Use% Mounted on
overlay         51572172 5797112  43640736  12% /
tmpfs              65536       0     65536   0% /dev
tmpfs            1021680       0   1021680   0% /sys/fs/cgroup
/dev/vda1       51572172 5797112  43640736  12% /etc/hosts
shm                65536       0     65536   0% /dev/shm
tmpfs            1021680      12   1021668   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            1021680       0   1021680   0% /proc/acpi
tmpfs            1021680       0   1021680   0% /sys/firmware
pavan@p1: kubectl exec mysql-deployment-95f7dd544-mmjv9 -- df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          50G  5.6G   42G  12% /
tmpfs            64M     0   64M   0% /dev
tmpfs           998M     0  998M   0% /sys/fs/cgroup
/dev/vda1        50G  5.6G   42G  12% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           998M   12K  998M   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           998M     0  998M   0% /proc/acpi
tmpfs           998M     0  998M   0% /sys/firmware

I coudn't quite understand the o/p, i reqested for 10Gi and i don't see any mount with 10gi total capacity?

PV definition :

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/mysql"

PVC definition:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Deployment definition:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql-deployment
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: mysql-pod
  template:
    metadata:
      labels:
        app: mysql-pod
    spec:
      containers:
      - name: mysql-container
        image: mysql:5.7
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

P.s: Node has a capacity of 50G

EDIT 1:

PV describe:

pavan@p1: kubectl describe pv/mysql-pv-volume
Name:            mysql-pv-volume
Labels:          type=local
Annotations:     kubectl.kubernetes.io/last-applied-configuration:
                   {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"type":"local"},"name":"mysql-pv-volume"},"spec":{"acc...
                 pv.kubernetes.io/bound-by-controller: yes
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    manual
Status:          Bound
Claim:           default/mysql-pv-claim
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        10Gi
Node Affinity:   <none>
Message:         
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /mnt/mysql
    HostPathType:  
Events:            <none>

PVC describe:

pavan@p1: kubectl describe pvc/mysql-pv-claim
Name:          mysql-pv-claim
Namespace:     default
StorageClass:  manual
Status:        Bound
Volume:        mysql-pv-volume
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"mysql-pv-claim","namespace":"default"},"spec":{"acc...
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    mysql-deployment-95f7dd544-mmjv9
Events:        <none>

Pod describe:

pavan@p1: kubectl describe pod/mysql-deployment-95f7dd544-mmjv9
Name:           mysql-deployment-95f7dd544-mmjv9
Namespace:      default
Priority:       0
Node:           pay0k-k8-dev-bytq/10.130.219.196
Start Time:     Mon, 09 Sep 2019 18:14:17 +0800
Labels:         app=mysql-pod
                pod-template-hash=95f7dd544
Annotations:    <none>
Status:         Running
IP:             10.244.0.123
Controlled By:  ReplicaSet/mysql-deployment-95f7dd544
Containers:
  mysql-container:
    Container ID:   docker://83f4730892fd6908ef3dfae3b9125d25cb7467d24df89323c43d3ab136376147
    Image:          mysql:5.7
    Image ID:       docker-pullable://mysql@sha256:1a121f2e7590f949b9ede7809395f209dd9910e331e8372e6682ba4bebcc020b
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 09 Sep 2019 18:14:19 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      MYSQL_DATABASE:       <set to the key 'mysql-database' of config map 'mysqldb'>          Optional: false
      MYSQL_ROOT_PASSWORD:  <set to the key 'mysql-root-password' in secret 'db-credentials'>  Optional: false
      MYSQL_USER:           <set to the key 'mysql-user' in secret 'db-credentials'>           Optional: false
      MYSQL_PASSWORD:       <set to the key 'mysql-password' in secret 'db-credentials'>       Optional: false
    Mounts:
      /var/lib/mysql from mysql-persistent-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-gbpxc (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  mysql-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-pv-claim
    ReadOnly:   false
  default-token-gbpxc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-gbpxc
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                        Message
  ----    ------     ----  ----                        -------
  Normal  Scheduled  41m   default-scheduler           Successfully assigned default/mysql-deployment-95f7dd544-mmjv9 to pay0k-k8-dev-bytq
  Normal  Pulled     41m   kubelet, pay0k-k8-dev-bytq  Container image "mysql:5.7" already present on machine
  Normal  Created    41m   kubelet, pay0k-k8-dev-bytq  Created container mysql-container
  Normal  Started    41m   kubelet, pay0k-k8-dev-bytq  Started container mysql-container

Can you specify your kubernetes provider (gke,awk, bare metal...)? — FL3SH, Sep 09 '19 at 10:37
Hi, this is quite strange that we do not see the mounted point(/var/lib/mysql) in your filesystem, can you describe your pod? ` kubectl describe pod $POD_NAME` — Suresh Vishnoi, Sep 09 '19 at 10:54

score 2 · Accepted Answer · answered Sep 09 '19 at 10:49

2

In your PV definition your specify hostPath, so your data is stored directly on worker - this is why you see 50GB and you are skipping this additional layer provided by cloud provider which create pv directly from pvc.

Keeping data directly on a node isn't good approach because node could be removed/replaced at any time.

You should use digital ocean pvs instead of you're manually defined, if you spin new pod on digital ocean's pv output from df should show 10GB.

answered Sep 09 '19 at 10:49

FL3SH

2,996
1
17
25

"In your PV definition your specify hostPath, so your data is stored directly on worker - this is why you see 50GB and you are skipping this additional layer provided by cloud provider which create pv directly from pvc." I don't understand, are you saying if we use hostpath, we wont be able to mount? – pa1 Sep 09 '19 at 12:53
If you are using `hostpath` data is stored on a worker, you can't use digital-ocean pv provisioner so you can't request a particular size of disk, because is it not supported for hostpath. – FL3SH Sep 09 '19 at 12:55
I understand and I want my data to be stored on the worker and not on the clouds PV. Interesting to learn we can't see the size of mount in case of host path. Any links to back this pls? – pa1 Sep 09 '19 at 12:59
Sorry, I never found out why hostpath work this way. – FL3SH Sep 09 '19 at 13:06

How to identify the storage space left in a PVC?

1 Answers1