21

I would like to be able to see all of the various things that happened to a kube cluster on a timeline, including when nodes were found to be dead, when new nodes were added, when pods crashed and when they were restarted.

So far the best that we have found is kubectl get event but that seems to have a few limitations:

  • it doesn't go back in time that far (I'm not sure how far it goes back. A day?)
  • it combines similar events and orders the resulting list by the time of the latest event in each group. This makes it impossible to know what happened during some time range since events in that range may have been combined with later events outside the range.

One idea that I have is to write a pod that will use the API to watch the stream of events and log them to a file. This would let us control retention and it seems that events that occur while we are watching will not be combined, solving the second problem as well.

What are other people doing about this?

malcolm
  • 311
  • 1
  • 2
  • 5
  • Welcome to Stack Overflow :-) Feel free to take the [tour] to learn more about this community – Breeze Mar 31 '16 at 23:32
  • This sounds like a great question to ask in the [Kubernetes special interest group for cluster operations](https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-ops). – Robert Bailey Apr 01 '16 at 16:30
  • https://stackoverflow.com/questions/45310287/kubernetes-sort-pods-by-age/73614140 (my answer: `kubectl alpha events` https://stackoverflow.com/a/73614140/67824) – Ohad Schneider Sep 05 '22 at 19:55

3 Answers3

28

In a pinch, it might be possible to figure out what happened earlier from various log, especially the kubelet logs?

Saving events

Beni Cherniavsky-Paskin
  • 9,483
  • 2
  • 50
  • 58
  • 1
    How can I change `event-ttl` for an already launched cluster, or how to specify `event-ttl` before creating a cluster? For example if I would like all my events in a cluster to be available for 24 hours? Is there a command for that? Thanks in advance! (I am running k8s on a GKE) – MtDersvan Aug 27 '17 at 01:07
  • 1
    No idea, deserves a question => I see you opened https://stackoverflow.com/questions/45904245/how-to-change-time-to-live-of-gke-cluster-events-from-default-1h0m0s – Beni Cherniavsky-Paskin Aug 28 '17 at 09:14
0

Have you checked out the pod specific events tab in the Dashboard?

Some events from a cluster I have running in GKE:

enter image description here

Ryan Cox
  • 4,993
  • 2
  • 25
  • 18
  • Ah. I should have clarified that I'm using kubernetes on AWS so I'm limited to what I can get from the command line or API. – malcolm Apr 01 '16 at 17:21
  • 1
    Dashboard is a cluster add on. You can run it anywhere. Source is on github. See these install instructions: http://kubernetes.io/docs/user-guide/ui/#accessing-the-dashboard – Ryan Cox Apr 01 '16 at 19:38
  • 4
    Oh, neat. I hadn't noticed that. Based on your screenshot, though, it seems to have similar problems to `kubectl get events` in that it dedups events. My fear is that it just pulls from `get events` and doesn't go back any further either. – malcolm Apr 05 '16 at 19:00
  • 2
    Dashboard does not do any magic, it uses the API and the events will be the same as from kubectl. – Oliver Mar 02 '20 at 22:07
0

kubernetes/heapster can persist event to gcl and influxdb, but for now there is no api to access stored data

ZhiQiang Fan
  • 986
  • 6
  • 7