Why are there many exited containers in Kubernetes after some times?

Question

Sometimes when I run docker ps -a, I see about 5-10 containers in Exited status. For example, after one or two hours I see them.

Why are there such these exited containers and why are they in Exited status?

9f6bd4fa5a05   8522d622299c             "/opt/bin/flanneld -…"   About a minute ago   Exited (1) About a minute ago             k8s_kube-flannel_kube-flannel-ds-x8s8v_kube-system_024c58c6-9cc2-4c2a-b0d3-12a49101a57b_3
7203f2beaaef   8522d622299c             "cp -f /etc/kube-fla…"   2 minutes ago        Exited (0) About a minute ago             k8s_install-cni_kube-flannel-ds-x8s8v_kube-system_024c58c6-9cc2-4c2a-b0d3-12a49101a57b_1
e4ade2e2617c   09708983cc37             "kube-controller-man…"   2 minutes ago        Exited (2) 2 minutes ago                  k8s_kube-controller-manager_kube-controller-manager-localhost_kube-system_00f8734ea34e7389b55cb740d1fcf000_2
1a3762bfa98c   k8s.gcr.io/pause:3.4.1   "/pause"                 2 minutes ago        Exited (0) 2 minutes ago                  k8s_POD_kube-controller-manager-localhost_kube-system_00f8734ea34e7389b55cb740d1fcf000_2
65555c2db5c6   nginx                    "/docker-entrypoint.…"   5 hours ago          Exited (0) 2 minutes ago                  k8s_task-pv-container_task-pv-pod_default_8bfa648a-ddb0-4531-8984-81d4ce8db444_0
a6b99bcc020a   k8s.gcr.io/pause:3.4.1   "/pause"                 5 hours ago          Exited (0) 2 minutes ago                  k8s_POD_task-pv-pod_default_8bfa648a-ddb0-4531-8984-81d4ce8db444_0
b625c68f4fce   296a6d5035e2             "/coredns -conf /etc…"   6 hours ago          Exited (0) 2 minutes ago                  k8s_coredns_coredns-558bd4d5db-c88d8_kube-system_6170d444-f992-4163-aff0-480cd1ed1ce4_1
7897a49ca8c6   296a6d5035e2             "/coredns -conf /etc…"   6 hours ago          Exited (0) 2 minutes ago                  k8s_coredns_coredns-558bd4d5db-z8kfs_kube-system_b34f29d6-bd72-40ba-a4ba-102e71965f98_1
9b7cac87b4bb   k8s.gcr.io/pause:3.4.1   "/pause"                 6 hours ago          Exited (0) 2 minutes ago                  k8s_POD_coredns-558bd4d5db-z8kfs_kube-system_b34f29d6-bd72-40ba-a4ba-102e71965f98_10
02661c9b507e   k8s.gcr.io/pause:3.4.1   "/pause"                 6 hours ago          Exited (0) 2 minutes ago                  k8s_POD_coredns-558bd4d5db-c88d8_kube-system_6170d444-f992-4163-aff0-480cd1ed1ce4_10
0f31cf13fc3a   38ddd85fe90e             "/usr/local/bin/kube…"   6 hours ago          Exited (2) 2 minutes ago                  k8s_kube-proxy_kube-proxy-s9rf6_kube-system_ff2a1ef4-7316-47c4-8584-aac1a1cd63d5_1
91a53859a423   k8s.gcr.io/pause:3.4.1   "/pause"                 6 hours ago          Exited (0) 2 minutes ago                  k8s_POD_kube-proxy-s9rf6_kube-system_ff2a1ef4-7316-47c4-8584-aac1a1cd63d5_1
a4833a1301a4   4d217480042e             "kube-apiserver --ad…"   6 hours ago          Exited (137) 2 minutes ago                k8s_kube-apiserver_kube-apiserver-localhost_kube-system_bbc60c89c6c6215aca670678072d707c_1
e4d76060736a   k8s.gcr.io/pause:3.4.1   "/pause"                 6 hours ago          Exited (0) 2 minutes ago                  k8s_POD_kube-apiserver-localhost_kube-system_bbc60c89c6c6215aca670678072d707c_1
7c7f09ab3633   0369cf4303ff             "etcd --advertise-cl…"   6 hours ago          Exited (0) 2 minutes ago                  k8s_etcd_etcd-localhost_kube-system_146e48c8e9d3b464d1f47c594a8becc8_1
0204740e75b0   k8s.gcr.io/pause:3.4.1   "/pause"                 6 hours ago          Exited (0) 2 minutes ago                  k8s_POD_etcd-localhost_kube-system_146e48c8e9d3b464d1f47c594a8becc8_1
8631dd16c767   62ad3129eca8             "kube-scheduler --au…"   6 hours ago          Exited (2) 2 minutes ago                  k8s_kube-scheduler_kube-scheduler-localhost_kube-system_7d7118d190aeaaf065b4e86b4982f05f_1
a59ee59475cd   k8s.gcr.io/pause:3.4.1   "/pause"                 6 hours ago          Exited (0) 2 minutes ago                  k8s_POD_kube-scheduler-localhost_kube-system_7d7118d190aeaaf065b4e86b4982f05f_1

The log of one of them:

docker logs k8s_kube-proxy_kube-proxy-s9rf6_kube-system_ff2a1ef4-7316-47c4-8584-aac1a1cd63d5_1
I0622 09:48:20.971080       1 node.go:172] Successfully retrieved node IP: 192.168.43.85
I0622 09:48:20.971170       1 server_others.go:140] Detected node IP 192.168.43.85
W0622 09:48:20.971200       1 server_others.go:592] Unknown proxy mode "", assuming iptables proxy
I0622 09:48:25.066916       1 server_others.go:206] kube-proxy running in dual-stack mode, IPv4-primary
I0622 09:48:25.067112       1 server_others.go:212] Using iptables Proxier.
I0622 09:48:25.067154       1 server_others.go:219] creating dualStackProxier for iptables.
W0622 09:48:25.067185       1 server_others.go:506] detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6
I0622 09:48:25.187175       1 server.go:643] Version: v1.21.0
I0622 09:48:25.407303       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0622 09:48:25.407351       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0622 09:48:25.407727       1 conntrack.go:83] Setting conntrack hashsize to 32768
I0622 09:48:25.477081       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0622 09:48:25.477170       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0622 09:48:25.517411       1 config.go:315] Starting service config controller
I0622 09:48:25.517679       1 config.go:224] Starting endpoint slice config controller
I0622 09:48:25.531370       1 shared_informer.go:240] Waiting for caches to sync for service config
I0622 09:48:25.548740       1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
W0622 09:48:25.665718       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 09:48:25.667197       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
I0622 09:48:25.731527       1 shared_informer.go:247] Caches are synced for service config 
I0622 09:48:25.753275       1 shared_informer.go:247] Caches are synced for endpoint slice config 
W0622 09:54:21.682298       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:01:18.747213       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:10:27.734119       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:18:50.736315       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:28:33.903205       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:36:05.938074       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:43:23.940703       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:49:20.951579       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 10:58:54.953921       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 11:08:28.961630       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 11:14:26.988498       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 11:20:12.010192       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 11:29:09.023270       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:00:28.838837       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:05:46.840726       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:10:50.843578       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:19:07.846779       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:26:46.848443       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:34:57.853658       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:42:00.887206       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:50:03.935106       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 14:57:18.937677       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 15:03:41.955870       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 15:11:26.957845       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W0622 15:17:51.979285       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice

docker logs k8s_kube-flannel_kube-flannel-ds-x8s8v_kube-system_024c58c6-9cc2-4c2a-b0d3-12a49101a57b_3
I0622 15:19:21.070951       1 main.go:520] Determining IP address of default interface
I0622 15:19:21.071786       1 main.go:533] Using interface with name enp0s3 and address 192.168.43.85
I0622 15:19:21.073223       1 main.go:550] Defaulting external address to interface address (192.168.43.85)
W0622 15:19:21.073336       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
E0622 15:19:31.157536       1 main.go:251] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-x8s8v': Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-x8s8v": net/http: TLS handshake timeout

Do a kubernetes describe pod to the evicted pods and there will be information in the events, more info https://stackoverflow.com/questions/51931113/kubectl-get-events-only-for-a-pod — paltaa, Jun 22 '21 at 15:57

score 1 · Accepted Answer · answered Jun 23 '21 at 06:27

1

You are using deprecated API:

discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice

Based on this documentation:

EndpointSlice

The discovery.k8s.io/v1beta1 API version of EndpointSlice will no longer be served in v1.25.

Migrate manifests and API clients to use the discovery.k8s.io/v1 API version, available since v1.21.

All existing persisted objects are accessible via the new API

Notable changes in discovery.k8s.io/v1: - use per Endpoint nodeName field instead of deprecated topology["kubernetes.io/hostname"] field - use per Endpoint zone field instead of deprecated topology["topology.kubernetes.io/zone"] field - topology is replaced with the deprecatedTopology field which is not writable in v1

You need to find all deprecated API and migrate to supported versions.

answered Jun 23 '21 at 06:27

Mikołaj Głodziak

4,775
7
28

I have installed Kubernetes via the docs of `k8s.io` And I don't see any v1.25 in https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG – Saeed Jun 23 '21 at 06:58
[Here](https://kubernetes.io/blog/2021/04/08/kubernetes-1-21-release-announcement/#major-changes) you can find changes in 1.21 version. It also explains where version 1.25 came from. You can also find more detailed explanation [here](https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/2433-topology-aware-hints/README.md). – Mikołaj Głodziak Jun 23 '21 at 07:27
This message means that as of version 1.21 your API is deprecated and as of (future) 1.25 it will not be available at all. You were given instructions on how to use the supported API for version 1.21. – Mikołaj Głodziak Jun 23 '21 at 07:29

Why are there many exited containers in Kubernetes after some times?

1 Answers1

EndpointSlice