1

I'm troubleshooting liveness probe failures. I'm able to extract specific entries from k8s event using this approach

k get events --sort-by=.metadata.creationTimestamp | grep Liveness

I'd like to get only the pod(s) that causing the issue.
I'm considering to pipe a cut, but I'm not sure which delimiter should I use to get the specific column.

Where I can find the delimiter related to that specific k8s resource(Events) used to printout the kubectl output?

Any other suggestion is appreciated

UPDATE so far these are the best options (w/o using extra tools) satisfying my specific needs:

k get events -o jsonpath='{range .items[*]}{.involvedObject.name}/{.involvedObject.namespace}: {.message}{"\n"}{end}' | grep Liveness

k get events -o custom-columns=POD:.involvedObject.name,NS:.involvedObject.namespace,MGG:.message | grep Liveness
Crixo
  • 3,060
  • 1
  • 24
  • 32

2 Answers2

2

there is a feature in kubernetes called jsonpath

validate if your jsonpath is correct with this online tool: https://jsonpath.com/

easily go through json keys with this online tool, so you needn't manually type the key names any more): http://jsonpathfinder.com/

so your command will be:

k get events --sort-by=.metadata.creationTimestamp --jsonpath '{ .xxxxxx }'

enter image description here enter image description here

BMW
  • 42,880
  • 12
  • 99
  • 116
  • Thanks @BMW for suggesting the tool, it sounds very useful, I'll check it out. Yes, I'm aware about jsonpath but can I also use it also for filtering purpose such as I'm doing w/ "grep Liveness"? I understood jsonpath was more for extracting/querying fields while field-selector should be the way to filter data (but it does not apply to any resource field - where can I found the supported fields x each k8s resource?) – Crixo Jan 07 '20 at 09:41
  • could you provide some sample codes for me to reference? I run the `events` command locally, but didn't see this key `Liveness`. Can't help more. – BMW Jan 07 '20 at 10:53
  • you have to create a pod w/ a failing livenessProbe so you'll get the string within the k8s event as describe by @rebello post here above – Crixo Jan 07 '20 at 11:11
1

Jsonpath is a little bit limited to be used with filter and conditions, maybe for your case jq will be more suitable.

I did a test using jq to filter the Output of my probe: I've tested using the yaml from this link

The message of probe failure from this pod is:

Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory

and the path for this message in json is .items[*].message

Using jq I can filter only message that contains "Liveness probe failed": and show the pod name:

k get events --sort-by=.metadata.creationTimestamp -o json | jq -c '.items[] | select(.message|test("Liveness probe failed")) | .metadata.name'

The output is: 

"liveness-exec.15e791c17b80a3c1"

You can use jq to format the message in order to get a more helpful output, with pod details.

Try to look this references links:

./jq

filter array based on values

I hope it helps!

Mr.KoopaKiller
  • 3,665
  • 10
  • 21
  • That's great but I wonder if it can be accomplished using standard k8s tools and standard linux tool (eg. grep, cut..) – Crixo Jan 07 '20 at 11:13