coredns can't start when using crio and with selinux on

Question

I know this question is asked many times, but all about docker, this time is crio.

CentOS Linux release 7.6
CRI-O Version: 1.16.1
Kubernetes: v1.16.3
KubeAdm: v1.16.3

CoreDNS pods are in Error/CrashLoopBackOff state, and audit.log shows selinux prevents CoreDNS to read from /var/lib/kubelet/container_id/volumes/

type=AVC msg=audit(1576203392.727:1431): avc: denied { read } for pid=15866 comm="coredns" name="Corefile" dev="dm-0" ino=35369330 scontext=system_u:system_r:container_t:s0:c307,c586 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1

type=AVC msg=audit(1576203392.727:1431): avc: denied { open } for pid=15866 comm="coredns" path="/etc/coredns/..2019_12_13_02_13_30.965446608/Corefile" dev="dm-0" ino=35369330 scontext=system_u:system_r:container_t:s0:c307,c586 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1

type=AVC msg=audit(1576203393.049:1432): avc: denied { open } for pid=15866 comm="coredns" path="/var/run/secrets/kubernetes.io/serviceaccount/..2019_12_13_02_13_30.605147375/token" dev="tmpfs" ino=124481 scontext=system_u:system_r:container_t:s0:c307,c586 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file permissive=1

if I use docker newer than 1.7, it works fine, I assume this may related with the patch of mounting volume with z/Z option.

I can add policy like underneath, but it will compromise security.

module coredns 0.1;

require {
  type tmpfs_t;
  type container_t;
  type var_lib_t;

  class file { open read };
}

allow container_t tmpfs_t:file open;
allow container_t var_lib_t:file { open read };

any better solution exists? just like docker, with a little efforts and don't compromise security.

How did you deploy your cluster? – Dawid Kruk Dec 24 '19 at 09:23 — Dawid Kruk, Dec 24 '19 at 09:23
using kubeadm, I updated the question – Cyron Dec 25 '19 at 06:08 — Cyron, Dec 25 '19 at 06:08

score 2 · Answer 1 · answered Dec 28 '19 at 11:41

On the host do the following

chcon -R -t container_file_t /var/lib/kubelet/container_id/volumes

This will change the label on the host volumes to be accessible by the containers SELinux label.

I do not know of a good way to handle the passing in of secrets. But adding the

allow container_t tmpfs_t:file open;

Would be probably best.

In OpenShift these are all handled Automatically, I believe. Although I don't work at that level of the stack.

score 2 · Answer 2 · answered Jan 06 '20 at 11:35

I've looked into it and it seems that the problem lays in kubelet version. Let me elaborate on that:

SELinux Volumes not relabeled in 1.16 - this link is providing more details about the issue.

I tried to reproduce this coredns issue on different versions of Kubernetes.

Issue shows on version 1.16 and newer. It seems to work properly with SELinux enabled on the version 1.15.6

For this to work you will need working CentOS and CRI-O environment.

CRI-O version:

Version:  0.1.0
RuntimeName:  cri-o
RuntimeVersion:  1.16.2
RuntimeApiVersion:  v1alpha1

To deploy this insfrastructure I followed this site for the most part: KubeVirt

Kubernetes v1.15.7

Steps to reproduce:

Disable SELinux and restart machine:
- $ setenforce 0
- $ sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config
- $ reboot
Check if SELinux is disabled by invoking command: $ sestatus
Install packages with $ yum install INSERT_PACKAGES_BELOW
- kubelet-1.15.7-0.x86_64
- kubeadm-1.15.7-0.x86_64
- kubectl-1.15.7-0.x86_64
Initialize Kubernetes cluster with following command $ kubeadm init --pod-network-cidr=10.244.0.0/16
Wait for cluster to initialize correctly and follow kubeadm instructions to connect to cluster
Apply Flannel CNI $ kubectl apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml

Check if coredns pods are running correctly with command: $ kubectl get pods -A

It should give similar output to that:

NAMESPACE     NAME                                         READY   STATUS    RESTARTS   AGE
kube-system   coredns-5c98db65d4-2c7lt                     1/1     Running   2          7m59s
kube-system   coredns-5c98db65d4-5dp9s                     1/1     Running   2          7m59s
kube-system   etcd-centos-kube-master                      1/1     Running   2          7m20s
kube-system   kube-apiserver-centos-kube-master            1/1     Running   2          7m4s
kube-system   kube-controller-manager-centos-kube-master   1/1     Running   2          6m55s
kube-system   kube-flannel-ds-amd64-mzh27                  1/1     Running   2          7m14s
kube-system   kube-proxy-bqll8                             1/1     Running   2          7m58s
kube-system   kube-scheduler-centos-kube-master            1/1     Running   2          6m58s

Coredns pods in kubernetes cluster with SELinux disabled are working properly.

Enable SELinux:

From root account invoke commands to enable SELinux and restart the machine:

$ setenforce 1
$ sed -i s/^SELINUX=.*$/SELINUX=enforcing/ /etc/selinux/config
$ reboot

Check if coredns pods are running correctly. They should not get crashloopbackoff error when running: kubectl get pods -A

Kubernetes v1.16.4

Steps to reproduce:

Run $ kubeadm reset if coming from another another version
Remove old Kubernetes packages with $ yum remove OLD_PACKAGES
Disable SELinux and restart machine:
- $ setenforce 0
- $ sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config
- $ reboot
Check if SELinux is disabled by invoking command: $ sestatus
Install packages with $ yum install INSERT_PACKAGES_BELOW
- kubelet-1.16.4-0.x86_64
- kubeadm-1.16.4-0.x86_64
- kubectl-1.16.4-0.x86_64
Initialize Kubernetes cluster with following command $ kubeadm init --pod-network-cidr=10.244.0.0/16
Wait for cluster to initialize correctly and follow kubeadm instructions to connect to cluster
Apply Flannel CNI $ kubectl apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml

Check if coredns pods are running correctly with command: $ kubectl get pods -A

It should give similar output to that:

NAMESPACE     NAME                                         READY   STATUS             RESTARTS   AGE
kube-system   coredns-5644d7b6d9-fgbkl                     1/1     Running            1          13m
kube-system   coredns-5644d7b6d9-x6h4l                     1/1     Running            1          13m
kube-system   etcd-centos-kube-master                      1/1     Running            1          12m
kube-system   kube-apiserver-centos-kube-master            1/1     Running            1          12m
kube-system   kube-controller-manager-centos-kube-master   1/1     Running            1          12m
kube-system   kube-proxy-v52ls                             1/1     Running            1          13m
kube-system   kube-scheduler-centos-kube-master            1/1     Running            1          12m

Enable SELinux:

From root account invoke commands to enable SELinux and restart the machine:

$ setenforce 1
$ sed -i s/^SELINUX=.*$/SELINUX=enforcing/ /etc/selinux/config
$ reboot

After reboot coredns pods should enter crashloopbackoff state as shown below:

NAMESPACE     NAME                                         READY   STATUS             RESTARTS   AGE
kube-system   coredns-5644d7b6d9-fgbkl                     0/1     CrashLoopBackOff   25         113m
kube-system   coredns-5644d7b6d9-x6h4l                     0/1     CrashLoopBackOff   25         113m
kube-system   etcd-centos-kube-master                      1/1     Running            1          112m
kube-system   kube-apiserver-centos-kube-master            1/1     Running            1          112m
kube-system   kube-controller-manager-centos-kube-master   1/1     Running            1          112m
kube-system   kube-proxy-v52ls                             1/1     Running            1          113m
kube-system   kube-scheduler-centos-kube-master            1/1     Running            1          112m

Logs from the pod coredns-5644d7b6d9-fgbkl show:

plugin/kubernetes: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied

I have test 1.16.5 and 1.16.6, seems the issue not solved. – Cyron Jan 23 '20 at 01:24 — Cyron, Jan 23 '20 at 01:24
@Cyron I've tested it on 1.17.0 and it didn't work either. – Dawid Kruk Jan 23 '20 at 07:13 — Dawid Kruk, Jan 23 '20 at 07:13

score 0 · Answer 3 · answered Jan 07 '20 at 01:06

0

I did it by the command underneath.

semanage fcontext -a -t container_file_t "/var/lib/kubelet/pods/pod_id/volumes(/.*)?"
restorecon -R -v /var/lib/kubelet/pods/pod_id/

answered Jan 07 '20 at 01:06

Cyron

41
4

coredns can't start when using crio and with selinux on

3 Answers3

Kubernetes v1.15.7

Kubernetes v1.16.4

Linked