get one of new pod name after kubectl rollout status | kubectl get pods --field-selector returns Terminating pods

Question

The problem is that after deploying new deployment like

kubectl apply -f deployment.yml

(let's say deployment with one replica)

Kubernetes will create second pod and shutdown the previous one - ok so far.

But immediately after kubectl apply I would like to detect in CI/CD if deployment was successful and in any case (no matter if rollout succeeded or failed) fetch log from one of newly deployed pod in order to provide as much as possible information in CI/CD log to determine what went wrong.

So I'm using

kubectl rollout status deployment deployment-name

which is waiting for deployment to rollout. Immediately after though you will end up with two pods, one in status "Running" and another "Terminating".

Now the problematic part: Normally I would use method like

kubectl get pods --selector=app=deployment-name --output=jsonpath='{.items[*].metadata.name}' --field-selector=status.phase=Running

but unfortunately it will return names of both pods ("Running" and "Terminating") separated with space.

Now I've tried also

kubectl get pods --selector=app=deployment-name --output=jsonpath='{.items[*].metadata.name}' --field-selector=status.phase=Running,status.phase!=Terminating

according to documentation:

but for some reason this will return exactly the same result, both pods running and terminating.

The quesiton is:

How to properly exclude TERMINATING pods from the result?

Answer 1

Known issue since 2018. Havent resolved yet.

Please refer to Kubectl returns pods in various states when only those in Running state are selected via --field-selector, or -o jsonpath for more details.

In short: There is NO normal, SHORT, adequate one-line command that would do what you want.

Reasons behind that:

This looks like not a bug. kubectl get pods output STATUS column not shows status.phase. kubectl get pods displays table format of PodList object and uses status.containerStatuses states to display STATUS column data.

Pod phase valid states are pending, running, succeeded, failed and unknown. From this document 'Running' state depends on 'restartPolicy'.

Jsonpath applies on 'PodList' object, not on kubectl output. status.containerStatuses gives whether pod containers running or not. Field selectors vary by Kubernetes resource types. status.containerStatuses not supported in pods field selectors.

Workarounds:

1. kubectl get pods | grep Running

2. kubectl get pods -o jsonpath='{.items[*].status.containerStatuses[*].state.running},{.items[*].metadata.name}' --field-selector=status.phase==Running | sed 's/ /\n/' | grep startedAt | awk -F',' '{print $2}'

3. (source) kubectl get pods --field-selector=status.phase=Running --template {{range .items}}{{ if not .metadata.deletionTimestamp }}{{.metadata.name}}{{"\n"}}{{end}}{{end}}

Answer 2

from the command line you can use commands like

grep

to catch what you like or you can do what I do and just look at all the pods with

watch 

command to see the changes and look at the errors from a separate terminal.

What I use for these kind of situations:

watch -n0.5 " kubectl get pods | grep -v 'Running' "

to catch pods with non Running status (Completed, OOMKilled, Pending, etc.) then from a separate terminal

kubectl logs podName && kubectl describe pods podName

I hope I was a little help.

Even for getting the latest revision number I use

kubectl rollout history deployment.v1.apps/deploymentName | tail -2 | head -1 | awk '{print $1}'

But maybe you can use

kubectl get pods  --sort-by=.metadata.creationTimestamp 

and then take the last one so finally maybe, you can use

kubectl logs -c [containerName] $(kubectl get pods  --sort-by=.metadata.creationTimestamp | tail -1 | awk '{print$1}') 

this can give me the latest created pod and its logs