mkdir /mnt/data: read-only file system Back-off restarting failed postgres container

Question

I'm new to Kubernetes, I tried to apply yaml file to create Postgres in GKE, I'm getting error as "Error: failed to start container "postgres": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system Back-off restarting failed container.

I thinki need to give permsions as RWX , when i tried to Login to pod i.e inside container..It is not allowing to login. ANyone please help me !!.

This is my Yaml file for Postgres:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: postgres
    spec:
      selector:
       matchLabels:
        app: postgres
      replicas: 1
      template:
        metadata:
          labels:
            app: postgres
        spec:
          containers:
            - name: postgres
              image: postgres:latest
              imagePullPolicy: "IfNotPresent"
              envFrom:
                - configMapRef:
                    name: postgres-config
              volumeMounts:
                - mountPath: /var/lib/postgresql/data
                  name: postgredb
          volumes:
            - name: postgredb
              persistentVolumeClaim:
                claimName: postgres-pv-claim

---
    kind: PersistentVolume
    apiVersion: v1
    metadata:
      name: postgres-pv-volume
      labels:
        type: local
        app: postgres
    spec:
      storageClassName: manual
      capacity:
        storage: 5Gi
      accessModes:
        - ReadWriteMany
      hostPath:
        path: "/mnt/data"
---
   

     kind: PersistentVolumeClaim
        apiVersion: v1
        metadata:
          name: postgres-pv-claim
          labels:
            app: postgres
        spec:
          storageClassName: manual
          accessModes:
            - ReadWriteMany
          resources:
            requests:
              storage: 5Gi
        
    ---
        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: postgres-config
          labels:
            app: postgres
        data:
          POSTGRES_DB: postgresdb
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: root
    ---
        apiVersion: v1
        kind: Service
        metadata:
          name: postgres
          labels:
            app: postgres
        spec:
           ports:
            - name: postgres
              port: 5432
              nodePort: 30432
           type: NodePort
           selector:
            app: postgres

I assume you used [this](https://severalnines.com/database-blog/using-kubernetes-deploy-postgresql) tutorial. You are using `type: local` and you want to use `/mnt/data` which on GKE is `ReadOnly`. You can use `path: /var/lib/test` I will write an answer for you in a couple of minutes. — PjoterS, Jun 25 '20 at 17:06

PjoterS · Accepted Answer · 2020-06-26T08:30:07.703

In your Persistent Volume you are using type: local which means that you want to create directory in /mnt. Local also do not support dynamic volume provisioning. If you will SSH to any of your nodes you will find that this folder is ReadOnly file system.

/mnt $ mkdir something mkdir: cannot create directory ‘something’: Read-only file system

As fastest workaround, you just could change in your PV YAML

    - ReadWriteMany
  hostPath:
    path: /mnt/data

To:

    - ReadWriteMany
  hostPath:
    path: /var/lib/data

Example:

$ kubectl apply -f pv-pvc.yaml
persistentvolume/postgres-pv-volume created
persistentvolumeclaim/postgres-pv-claim created
$ kubectl apply -f pos.yaml
deployment.apps/postgres created
$ kubectl get po
NAME                        READY   STATUS    RESTARTS   AGE
postgres-65d9cbd495-pcqf5   1/1     Running   0          2s

$ kubectl exec -ti postgres-65d9cbd495-pcqf5 -- /bin/bash
root@postgres-65d9cbd495-pcqf5:/# cd /var/lib/postgresql/data
root@postgres-65d9cbd495-pcqf5:/var/lib/postgresql/data# ls
base    pg_commit_ts  pg_hba.conf    pg_logical    pg_notify    pg_serial     pg_stat      pg_subtrans  pg_twophase  pg_wal   postgresql.auto.conf  postmaster.opts
global  pg_dynshmem   pg_ident.conf  pg_multixact  pg_replslot  pg_snapshots  pg_stat_tmp  pg_tblspc    PG_VERSION   pg_xact  postgresql.conf       postmaster.pid
root@postgres-65d9cbd495-pcqf5:/var/lib/postgresql/data# echo "Hello from postgress pod" > data.txt
root@postgres-65d9cbd495-pcqf5:/var/lib/postgresql/data# cat data.txt
Hello from postgress pod

Now if you will SSH to the node which is hosting this pod, you will be able to reach this folder and files.

user@gke-cluster-1-default-pool-463f9615-gxhl ~ $ sudo su
gke-cluster-1-default-pool-463f9615-gxhl /home/user # cd /var/lib/data
gke-cluster-1-default-pool-463f9615-gxhl /var/lib/data # ls
PG_VERSION    pg_dynshmem    pg_notify     pg_stat_tmp  pg_xact
base          pg_hba.conf    pg_replslot   pg_subtrans  postgresql.auto.conf
data.txt      pg_ident.conf  pg_serial     pg_tblspc    postgresql.conf
global        pg_logical     pg_snapshots  pg_twophase  postmaster.opts
pg_commit_ts  pg_multixact   pg_stat       pg_wal       postmaster.pid
gke-cluster-1-default-pool-463f9615-gxhl /var/lib/data # cat data.txt 
Hello from postgress pod

EDIT

YAMLs Ive used.

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  labels:
    app: postgres
data:
  POSTGRES_DB: postgresdb
  POSTGRES_USER: postgres
  POSTGRES_PASSWORD: root
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  type: NodePort
  selector:
    app: postgres
  ports:
  - name: postgres
    port: 5432
    nodePort: 30432
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-pv-volume
  labels:
    app: postgres
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: /var/lib/data
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgres-pv-claim
  labels:
    app: postgres
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:latest
          imagePullPolicy: "IfNotPresent"
          envFrom:
            - configMapRef:
                name: postgres-config
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgredb
      volumes:
        - name: postgredb
          persistentVolumeClaim:
            claimName: postgres-pv-claim
            
configmap/postgres-config created
service/postgres created
persistentvolume/postgres-pv-volume created
persistentvolumeclaim/postgres-pv-claim created
deployment.apps/postgres created

$ kubectl get po
NAME                        READY   STATUS    RESTARTS   AGE
postgres-65d9cbd495-wxx4h   1/1     Running   0          19s

If i change path according to you specified getting error as "The PersistentVolume "postgres-pv-volume" is invalid: spec.persistentvolumesource: Forbidden: is immutable after creation" — ChinnapaReddy, Jun 25 '20 at 18:19
You must delete PV and PVC (if they wont vanish you will probably need redeploy postgress deployment also) as in some resources you cannot change values after creation. — PjoterS, Jun 25 '20 at 18:35
I have deleted and same error and also I deployed that in a new cluster again same error coming — ChinnapaReddy, Jun 26 '20 at 04:42
Which one error? `The PersistentVolume "postgres-pv-volume" is invalid: spec.persistentvolumesource: Forbidden: is immutable after creation` or `read-only file system` — PjoterS, Jun 26 '20 at 08:20
If you still have `Forbidden: is immutable after creation` error, please ehck if PersistentVolume and PersistentVolumeClaim was removed. `$ kubectl get cm,deploy,pv,pvc No resources found in default namespace.` Ive added YAMLs that I have used. Just to Confirm you are using GKE [Google Kubernetes Engine] not Kubeadm or Minikue on local machine? — PjoterS, Jun 26 '20 at 08:31
If you have any Kubernetes/Devops community means please share link to join or Invite me on Kanakanti98@gmail.com. Thanks in advance — ChinnapaReddy, Jun 27 '20 at 17:01
this does not work for me, I get: `Error: stat /var/lib/data: no such file or directory` — gtato, Apr 08 '22 at 15:51

score 0 · Answer 2 · answered Jan 05 '22 at 09:52

0

If you're working with GKE, just create PVC, it will self create PV which will work according to your need.

I Fixed my issue that way.

answered Jan 05 '22 at 09:52

Muhammad Jamee Ghouri

79
1
4

I tried this, but it didn't work for me. There was still an error on startup `read-only file system: unknown` when trying this. – fuzzi May 02 '22 at 23:17
Share the manifest file. – Muhammad Jamee Ghouri May 09 '22 at 07:41

mkdir /mnt/data: read-only file system Back-off restarting failed postgres container

2 Answers2

Linked