4

I setup my kubernetes cluster by kubeadm. at the last step. I need join my nodes to the master. So i copy the join command to the node to exec it and whitout err. Then i back to master to exec "kubectl get nodes". there just a master node and without worker node. So i check the worker node's kubelet log and it show the err like this:

2月 16 15:51:44 localhost.localdomain kubelet[938]: E0216 15:51:44.608258 938 controller.go:177] failed to update node lease, error: Operation cannot be fulfilled on leases.coordination.k8s.io "localhost.localdomain": the object has been modified; please apply your changes to the latest version and try again I have no ideal to how to solve it. by the way, i have two question, this is one of it. Another is about kube-apiserver: kubernetes v1.16.2 write: broken out I still haven't solved it here is the main step i setup the cluster

setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

systemctl stop firewalld && systemctl disable firewalld
iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT

swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

modprobe br_netfilter
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
sysctl -p

cat << EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf

yum install -y epel-release conntrack ipvsadm ipset jq sysstat curl iptables libseccomp  unzip lrzsz
yum install -y yum-utils device-mapper-persistent-data lvm2  ntp ntpdate

ntpdate ntp1.aliyun.com

yum install docker-ce -y
yum -y install  kubelet-1.17.0 kubeadm-1.17.0 kubectl-1.17.0  kubernetes-cni

systemctl enable docker
systemctl enable kubelet.service
systemctl start docker
systemctl start kubelet

kubeadm config images list

kubeadm config print init-defaults > kubeadm.conf

kubeadm init --kubernetes-version=v1.17.0 --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=172.16.5.150

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

then copy this to node to exec
kubeadm join 172.16.5.150:6443 --token 2yj4eu.lhdrdks0dykao9in \
    --discovery-token-ca-cert-hash sha256:8563b0c50c48e563cdd4ac4380206133535df8a020bc2a949ff66b5aa7722ea7

[root@localhost calico]# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                       READY   STATUS             RESTARTS   AGE   IP                NODE                    NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-5b644bc49c-8ht2m   1/1     Running            0          82m   192.168.102.134   localhost.localdomain   <none>           <none>
kube-system   calico-node-4887h                          1/1     Running            0          82m   172.16.5.152      localhost.localdomain   <none>           <none>
kube-system   coredns-6955765f44-9pjbp                   1/1     Running            0          9h    192.168.102.133   localhost.localdomain   <none>           <none>
kube-system   coredns-6955765f44-mlrdt                   0/1     CrashLoopBackOff   17         9h    192.168.102.132   localhost.localdomain   <none>           <none>
kube-system   kube-proxy-6fqwk                           1/1     Running            0          9h    172.16.5.151      localhost.localdomain   <none>           <none>
[root@localhost calico]# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                       READY   STATUS             RESTARTS   AGE   IP                NODE                    NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-5b644bc49c-8ht2m   0/1     Running            1          82m   192.168.102.137   localhost.localdomain   <none>           <none>
kube-system   calico-node-4887h                          1/1     Running            0          82m   172.16.5.159      localhost.localdomain   <none>           <none>
kube-system   coredns-6955765f44-9pjbp                   1/1     Running            16         9h    192.168.102.136   localhost.localdomain   <none>           <none>
kube-system   coredns-6955765f44-mlrdt                   0/1     CrashLoopBackOff   15         9h    192.168.102.135   localhost.localdomain   <none>           <none>
kube-system   kube-proxy-6fqwk                           1/1     Running            0          9h    172.16.5.151      localhost.localdomain   <none>           <none>
[root@localhost calico]# kubectl get nodes
NAME                    STATUS   ROLES    AGE   VERSION
localhost.localdomain   Ready    master   9h    v1.17.0
[root@localhost calico]#
Esc
  • 521
  • 13
  • 30
  • output of kubectl get pods -n kube-system? Also can you paste the join command that you ran ? At the end did you see a message "Node join complete" ? – Arghya Sadhu Feb 16 '20 at 08:44
  • @Arghya Sadhu here is the output and the join command [root@localhost .kube]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-6955765f44-9pjbp 0/1 Pending 0 3h38m coredns-6955765f44-mlrdt 0/1 Pending 0 3h38m kube-proxy-6fqwk 1/1 Running 0 3h38m [root@localhost .kube]# kubeadm join 172.16.5.150:6443 --token lkgzrz.i8z7i8vkcehlk4cs --discovery-token-ca-cert-hash sha256:641dfd5e25022152145e34ea0aeb4816ceb9d9c66c9e398145b55287116cafc9 – Esc Feb 16 '20 at 09:06
  • Next to the previous one. when i exec the join command on the worker node. the output is: ```Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.``` this is the last log of the output – Esc Feb 16 '20 at 09:10
  • why don't you add above details in the question itself with proper formatting – Arghya Sadhu Feb 16 '20 at 09:14
  • kube-system namespace should have few more pods for api sever, controller, scheduler etc.can you provide the output of kubectl get componentstatuses. Also which docs specifically you followed ? – Arghya Sadhu Feb 16 '20 at 10:17
  • @ Arghya Sadhu. sorry about the format. I am learing how to format the question and comment. [root@localhost .kube]# kubectl get componentstatuses NAME STATUS MESSAGE ERROR
    scheduler Healthy ok
    controller-manager Healthy ok
    etcd-0 Healthy {"health":"true"}
    I am in china. And in china. for the reason of the ISP Block.i can't visit google directly. So i follow the docs of the chinese docs. if necessary. i can paste the main step.     – Esc Feb 16 '20 at 11:29
  • yes provide the main steps...are you in public cloud or on Prem bare metal? – Arghya Sadhu Feb 16 '20 at 11:33
  • @Arghya Sadhu The comment limits the w ord count. and I am not good at format on stackoverflow. So may i get your email and send to your emai or here is my gmail: yinkaicool@gmail.com – Esc Feb 16 '20 at 11:50
  • You can edit your question and add the steps. – Arghya Sadhu Feb 16 '20 at 12:03
  • @Arghya Sadhu you can see in the question now – Esc Feb 16 '20 at 12:13
  • You need to install a pod network add on.Did you install a pod network add on? kubectl apply -f https://docs.projectcalico.org/v3.11/manifests/calico.yaml – Arghya Sadhu Feb 16 '20 at 12:21
  • @ Arghya Sadhu I install calico now. And the master is ready now. but i still can't get the worker nodes on master by command "kubectl get nodes" – Esc Feb 16 '20 at 13:39
  • Did you run the join command again ? Can you run it and what error you get ? – Arghya Sadhu Feb 16 '20 at 13:41
  • yes i run "kubeadm reset" on the worker node. then i exec the join command again. the output is as same as before like this: ``` [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ``` this is the last log of the output – Esc Feb 16 '20 at 14:31
  • And kubelet logs from both worker and master node? Also what pods you see kubectl get pods -n Kube-system – Arghya Sadhu Feb 16 '20 at 14:32
  • @ Arghya Sadhu.There is interesting. I run "docker ps" on the worker node and found that the calico component container has been dispatch to he worker node and is running up. – Esc Feb 16 '20 at 14:45
  • @ Arghya Sadhu I paste the output on the question. you can see there. – Esc Feb 16 '20 at 14:52
  • kubelet logs from worker and master both – Arghya Sadhu Feb 16 '20 at 15:28
  • update the question with the following details: exact `kubeadm init` command you issued, exact `kubeadm join` you issued, also, sicne coredns is failing, `kubectl describe pod ` and `kubectl logs `, `cat /etc/resolv.conf` on both master and the node, `host ` from the node machine. – morgwai Feb 17 '20 at 07:38
  • @morgwai thanks.it will take your suggestion. – Esc Feb 19 '20 at 16:04
  • @Esc does this issue was the same as https://stackoverflow.com/questions/60305724/service-kube-dns-is-invalid-spec-clusterip-invalid-value-10-10-0-10-fiel ?Or it's diffrent and still occurs? – PjoterS Feb 24 '20 at 13:27
  • @Esc Could you please answer the question above? – Wytrzymały Wiktor Mar 01 '21 at 14:04
  • @Esc Is your issue resolved?if yes, can you post the procedure you've followed as Solution and accept it. – Goli Nikitha Sep 23 '21 at 08:47
  • Is this issue is resolved ? If yes, can you post the procedure you've followed as solution and accept it. – Venkata Satya Karthik Varun Ku Mar 26 '22 at 12:22

0 Answers0