cannot mount existing data to persistent volume in kubernetes

Question

I am using clickhouse database and data are stored at /media/user/data/clickhouse and /media/user/data/clickhouse-server. When I run a docker container

$ docker run \
    --name local-clickhouse \
    --ulimit nofile=262144:262144 \
    -u 1000:1000 \
    -p 8123:8123 \
    -p 9000:9000 \
    -p 9009:9009 \
    -v /media/user/data/clickhouse:/var/lib/clickhouse \
    -v /media/user/data/clickhouse-server:/var/log/clickhouse-server \
    -dit clickhouse/clickhouse-server

I see the data and everything is fine. I am trying to run this in a pod using minikube with following persistent volume configs:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: host-pv-clickhouse
spec:
  capacity:
    storage: 4000Gi   
  volumeMode: Filesystem
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /media/user/data/clickhouse
    type: DirectoryOrCreate

and

apiVersion: v1
kind: PersistentVolume
metadata:
  name: host-pv-clickhouse-server
spec:
  capacity:
    storage: 4000Gi 
  volumeMode: Filesystem  
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /media/user/data/clickhouse-server
    type: DirectoryOrCreate

Additionally, I also have persistent volume claims:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: host-pvc-clickhouse-server
spec:
  volumeName: host-pv-clickhouse-server
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2000Gi

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: host-pvc-clickhouse
spec:
  volumeName: host-pv-clickhouse
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2000Gi

and finally service and deployment:

apiVersion: v1
kind: Service
metadata:
  name: clickhouse
spec:
  type: NodePort
  ports:
    - protocol: TCP
      name: tcp
      port: 9000
      targetPort: 9000
      nodePort: 30003
    - protocol: TCP
      name: http
      port: 8123
      targetPort: 8123
      nodePort: 30004
    - protocol: TCP
      name: interncomm
      port: 9009
      targetPort: 9009
      nodePort: 30005
  selector:
    app: clickhouse
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: clickhouse
  labels:
    app: clickhouse
spec:
  replicas: 1
  selector:
    matchLabels:
      app: clickhouse
  template:
    metadata:
      labels:
        app: clickhouse
    spec:
      containers:
        - name: clickhouse
          image: clickhouse/clickhouse-server:latest
          ports:
            - containerPort: 8123
              name: http
            - containerPort: 9000
              name: tcp
            - containerPort: 9009
              name: interncomm
          volumeMounts:
            - name: clickhouse-volume
              mountPath: /var/lib/clickhouse
            - name: clickhouse-server-volume
              mountPath: /var/log/clickhouse-server
      volumes:
        - name: clickhouse-volume
          persistentVolumeClaim: 
            claimName: host-pvc-clickhouse
        - name: clickhouse-server-volume
          persistentVolumeClaim: 
            claimName: host-pvc-clickhouse-server

When I run kubectl apply -f chdb_node.yaml it works and I can access the database via clickhouse's web gui. However, the data aren't there.

Any suggestions to how to fix this?

Does this answer your question? [Mount local directory into pod in minikube](https://stackoverflow.com/questions/48534980/mount-local-directory-into-pod-in-minikube) — Rick Rackow, Aug 11 '23 at 09:06
@RickRackow it still won't load the data when I use `hostPath` and `path` — MoneyBall, Aug 11 '23 at 09:14
I wouldn't necessarily pursue this path. Kubernetes is pretty complex to use it for only a single-system setup. If you're trying to eventually run this in a multi-host setup, you can't use a `hostPath` volume, and you'd need to figure out how to back up and restore the data from wherever the cluster keeps it. You should be able to do the same thing with your test Minikube setup. — David Maze, Aug 11 '23 at 10:24
@MoneyBall is your issue resolved..? revert back here if you are still facing some issues. — Kranthiveer Dontineni, Aug 15 '23 at 04:02

score 1 · Answer 1 · answered Aug 11 '23 at 12:24

Have you got a chance to go through this official kubernetes tutorial, also you are using type: DirectoryOrCreate which will create a directory if the original directory is not available.

Hence it is suggested to check whether the directories which you are trying to mount on the pod already exist on the node by using the below command

ls -lah /media/user/data/clickhouse

and 

ls -lah /media/user/data/clickhouse-server

This command also helps in checking whether there is any data available in these directories. In case data is not available in these directories, copy the data from your docker container or from the source directory to these directories and follow the official tutorial for mounting your persistent volumes.

score 1 · Accepted Answer · answered Aug 15 '23 at 20:27

Check first the data at /media/user/data/clickhouse and /media/user/data/clickhouse-server on your host machine indeed contains the expected ClickHouse data.

And remember that, when you use Minikube, it typically runs in a virtual machine (like VirtualBox). It means the hostPath you are providing is referencing the filesystem of the VM, not your actual host.
To use host filesystem in Minikube with hostPath, you should make sure that the directory is properly mounted into the Minikube VM, using minikube mount.

minikube mount /media/user/data:/media/user/data

That would mount the /media/user/data directory from your host into the Minikube VM at the same location. After doing this, your hostPath configuration should work as expected.

Warning: Your Docker run command specifies a user and group with UID and GID as 1000:1000. Ensure that the files and directories at /media/user/data/clickhouse and /media/user/data/clickhouse-server are owned by this UID and GID. If not, the ClickHouse server might not be able to read them.
In your Kubernetes pod, you did not specify the user or group to run the container. You might want to set the same user as you did with Docker using the securityContext:

spec:
    containers:
    - name: clickhouse
        ...
        securityContext:
        runAsUser: 1000
        runAsGroup: 1000

after mounting and changing the security context, I was able to see the data! Thank you! — MoneyBall, Aug 21 '23 at 23:52

cannot mount existing data to persistent volume in kubernetes

2 Answers2