minikube shared volume not shows files after sometime

Question

I have to share local .ssh directory content to pod. I search for hat and got answer from one of the post to share start as --mount-string.

$ minikube start --mount-string="$HOME/.ssh/:/ssh-directory" --mount
  minikube v1.9.2 on Darwin 10.14.6
✨  Using the docker driver based on existing profile
  Starting control plane node m01 in cluster minikube
  Pulling base image ...
  Restarting existing docker container for "minikube" ...
  Preparing Kubernetes v1.18.0 on Docker 19.03.2 ...
    ▪ kubeadm.pod-network-cidr=10.244.0.0/16
E0426 23:44:18.447396   80170 kubeadm.go:331] Overriding stale ClientConfig host https://127.0.0.1:32810 with https://127.0.0.1:32813
  Creating mount /Users/myhome/.ssh/:/ssh-directory ...
  Enabling addons: default-storageclass, storage-provisioner
  Done! kubectl is now configured to use "minikube"

❗  /usr/local/bin/kubectl is v1.15.5, which may be incompatible with Kubernetes v1.18.0.
  You can also use 'minikube kubectl -- get pods' to invoke a matching version

When I check the docker for the given Minikube, it return

$ docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS                                                                           NAMES
5ad64f642b63        gcr.io/k8s-minikube/kicbase:v0.0.8   "/usr/local/bin/entr…"   3 weeks ago         Up 45 seconds       127.0.0.1:32815->22/tcp, 127.0.0.1:32814->2376/tcp, 127.0.0.1:32813->8443/tcp   minikube

And check the .ssh directory content are there or not.

$ docker exec -it 5ad64f642b63 ls /ssh-directory
id_rsa  id_rsa.pub  known_hosts

I have deployment yml as

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-deployment
  labels:
    stack: api
    app: api-web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-web
  template:
    metadata:
      labels:
        app: api-web
    spec:
      containers:
        - name: api-web-pod
          image: tiangolo/uwsgi-nginx-flask 
          ports:
            - name: api-web-port
              containerPort: 80
          envFrom:
            - secretRef:
                name: api-secrets
          volumeMounts:
            - name: ssh-directory
              mountPath: /app/.ssh
      volumes:
        - name: ssh-directory
          hostPath:
            path: /ssh-directory/
            type: Directory

When it ran, it gives error for /ssh-directory.

$ kubectl describe pod/api-deployment-f65db9c6c-cwtvt
Name:           api-deployment-f65db9c6c-cwtvt
Namespace:      default
Priority:       0
Node:           minikube/172.17.0.2
Start Time:     Sat, 02 May 2020 23:07:51 -0500
Labels:         app=api-web
                pod-template-hash=f65db9c6c
Annotations:    <none>
Status:         Pending
IP:
Controlled By:  ReplicaSet/api-deployment-f65db9c6c
Containers:
  api-web-pod:
    Container ID:
    Image:          tiangolo/uwsgi-nginx-flask
    Image ID:
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      api-secrets  Secret  Optional: false
    Environment:      <none>
    Mounts:
      /app/.ssh from ssh-directory (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-9shz5 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  ssh-directory:
    Type:          HostPath (bare host directory volume)
    Path:          /ssh-directory/
    HostPathType:  Directory
  default-token-9shz5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-9shz5
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    <unknown>           default-scheduler  Successfully assigned default/api-deployment-f65db9c6c-cwtvt to minikube
  Warning  FailedMount  11m                 kubelet, minikube  Unable to attach or mount volumes: unmounted volumes=[ssh-directory], unattached volumes=[default-token-9shz5 ssh-directory]: timed out waiting for the condition
  Warning  FailedMount  2m13s (x4 over 9m)  kubelet, minikube  Unable to attach or mount volumes: unmounted volumes=[ssh-directory], unattached volumes=[ssh-directory default-token-9shz5]: timed out waiting for the condition
  Warning  FailedMount  62s (x14 over 13m)  kubelet, minikube  MountVolume.SetUp failed for volume "ssh-directory" : hostPath type check failed: /ssh-directory/ is not a directory

When I check the content of /ssh-directory in docker.

It gives IO error.

$ docker exec -it 5ad64f642b63 ls /ssh-directory
ls: cannot access '/ssh-directory': Input/output error

I know there are default mount points for Minikube. As mentioned in https://minikube.sigs.k8s.io/docs/handbook/mount/,

+------------+----------+---------------+----------------+
| Driver     |    OS    | HostFolder    | VM             |
+------------+----------+---------------+----------------+
| VirtualBox | Linux    | /home         |/hosthome       |
+------------+----------+---------------+----------------+
| VirtualBox | macOS    | /Users        |/Users          |
+------------+----------+---------------+----------------+
| VirtualBox | Windows  |C://Users      | /c/Users       |
+------------+----------+---------------+----------------+
|VMware Fusio|  macOS   |/Users         |/Users          |
+------------+----------+---------------+----------------+
| KVM        | Linux    | Unsupported.  |                |
+------------+----------+---------------+----------------+
| HyperKit   | Linux    | Unsupported   |(see NFS mounts)|  
+------------+----------+---------------+----------------+

But I installed minikube as brew install minikube and its set driver as docker.

$ cat ~/.minikube/config/config.json
{
    "driver": "docker"
}

There is no mapping for docker driver in mount point.

Initially, this directory has the files, but somehow, when I try to create the pod, it delete or something is wrong.

did you try to reference your hosthome directly in your yml file ? there is a special keyname for this "hosthome/.ssh" in your case have a look at the section Driver (bottom of the page) here : https://minikube.sigs.k8s.io/docs/handbook/mount/ — jossefaz, May 03 '20 at 05:05
if you do this you won't have to mount `--mount-string="$HOME/.ssh/:/ssh-directory"` in the minikube start — jossefaz, May 03 '20 at 05:10
In general you can't mount files like this with Kubernetes. (It might be possible for desktop installations like Minikube or Kind, but not for more typical clustered installations.) Can you put this key into a Secret instead, where it won't depend on filesystem content? — David Maze, May 03 '20 at 09:59
@DavidMaze I follow the answer in https://stackoverflow.com/questions/48534980/mount-local-directory-into-pod-in-minikube/48535001#48535001 to shared the local folder, — Nilesh, May 03 '20 at 16:24
@Nilesh : if you have MacOS, you can get to your home directory by calling the /Users folder (pre mounted for you by minikube) — jossefaz, May 03 '20 at 17:40
@yAzou I tried ```$ docker exec -it 5ad64f642b63 ls /User ls: cannot access '/User': No such file or directory```, but it gives error. — Nilesh, May 03 '20 at 19:32
@yAzou as per [this docs](https://kubernetes.io/docs/setup/learning-environment/minikube/#mounted-host-folders), there is `/Users` directory if we use `driver` as `VirtualBox`, but my driver is `docker`. And there is no mention of `docker` driver in it. — Nilesh, May 03 '20 at 19:37
@Nilesh if you use the predefined mounting : it is not with the docker command. It is as you said with minikube VB. You just have to change the hostpath in your yaml file from "/ssh-directory" to "Users/.shh". Then you just do Kubectl apply. — jossefaz, May 04 '20 at 04:08
@Nilesh can you try solution suggeted by 'yAzou' and let u know if it works. — acid_fuji, May 04 '20 at 12:28
@acid_fuji https://minikube.sigs.k8s.io/docs/handbook/mount/ suggested by `yAzou`, but there it not cover the driver as `docker`. Let me update my question. — Nilesh, May 04 '20 at 17:58
Sorry for not update the all details in first question, but now I updated with all driver and other information. — Nilesh, May 04 '20 at 18:07

score 2 · Accepted Answer · answered May 06 '20 at 11:22

While reproducing this on ubuntu I encountered the exact issue.

The directory was indeed looked like mounted but the files were missing which lead me to think that this is a general issue with mounting directories with docker driver.

There is open issue on github about the same problem ( mount directory empty ) and open feature request to mount host volumes into docker driver.

Inspecting minikube container shows no record of that mounted volume and confirms information mentioned in the github request that the only volume shared with host as of now is the one that mounts by default (that is /var/lib/docker/volumes/minikube/_data mounted into minikube's /var directory).

$ docker inspect minikube
"Mounts": [ 
  { 
    "Type": "volume", 
    "Name": "minikube", 
    "Source": "/var/lib/docker/volumes/minikube/_data", 
    "Destination": "/var", 
    "Driver": "local", 
    "Mode": "z", 
    "RW": true, 
    "Propagation": ""
  }

As the workaround you could copy your .ssh directory into the running minikube docker container with following command:

docker cp  $HOME/.ssh minikube:<DESIRED_DIRECTORY>

and then mount this desired directory into the pod.

Thanks, I might be prefer the ssh key in secrets. That way, there is no manual steps. — Nilesh, May 06 '20 at 23:53

minikube shared volume not shows files after sometime

1 Answers1