Pods stuck in pending state

Question

I am using the Kubernetes-client java client to create Deployments on a Kubernetes cluster. THis is the code

Deployment deployment = new DeploymentBuilder()
        .withNewMetadata()
        .withName("first-deployment")
        .endMetadata()
        .withNewSpec()
        .withReplicas(3)
        .withNewTemplate()
        .withNewMetadata()
        .addToLabels(namespaceID, "hello-world-example")
        .endMetadata()
        .withNewSpec()
        .addNewContainer()      
        .withName("nginx-one")
        .withImage("nginx")
        .addNewPort()
        .withContainerPort(80)
        .endPort()
        .withResources(resourceRequirements)
        .endContainer()
        .endSpec()
        .endTemplate()
        .endSpec()
        .build();
    deployment = client.extensions().deployments().inNamespace(namespace).create(deployment);

I add a3 min wait time and then test the status of the pod

PodList podList = client.pods().withLabel(namespaceID, "hello-world-example").list();
    System.out.println("Number of pods " + podList.getItems().size());
    for (Pod pod : podList.getItems()) {
        System.out.println("Name " + pod.getMetadata().getName() 
            + " Status " + pod.getStatus().getPhase() 
            + " Reason " + pod.getStatus().getReason()
        + " Containers " + pod.getSpec().getContainers().get(0).getResources().getLimits());

    }

This returns the following sttaus

Name first-deployment-2418943216-9915m Status Pending Reason null Containers null
Name first-deployment-2418943216-fnk21 Status Pending Reason null Containers null
Name first-deployment-2418943216-zb5hr Status Pending Reason null Containers null

However from the commandline if I get kubectl get pods --all-namespaces. It returns the pod state as running . Am I using the right API? what did I miss?

It's quite possible to get this if the remote hosts were still scheduling and spinning up the containers when your status call was invoked, but when you looked manually they had completed, so I *think* you've got the rough code correct and may simply need to wait a bit longer. The way you can tell is did the results of `kubectl get pods --all-namespaces` return the same pod names as your API call, just differing in status, reason, and container references? — heckj, Jan 19 '18 at 19:27
Yeah thats the thing. I waited ~ 4mins and the status was pending. I manually checked using kubectl and the [ods were running. I am not sure why the library gives a different result that the commandline kuectl — user_mda, Jan 19 '18 at 19:55
Possibly related to https://github.com/kubernetes/kubernetes/issues/29876 — mghicks, Jan 25 '18 at 22:35

score 2 · Answer 1 · answered Mar 24 '18 at 11:36

Maybe a better way to check this is to have a loop and sleep inside to loop and continuously keep checking the status until all pods are up are running. I had done something similar to check if all the required pods were up by checking the status. But you might also want to consider adding the liveness and readiness probe on the pods before you make such a check. There are additional details provided here.

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/

score 1 · Answer 2 · answered May 31 '18 at 01:54

Not sure if this question is still open, but I would suggest to use something like this: https://github.com/kubernetes/helm/blob/d790f7d843182f1c126cfa64989ffdd4e9583747/pkg/kube/wait.go#L174

It loops through deployments, and wait till max available from manifest.

Pods stuck in pending state

2 Answers2