I have services that has a "controllable" memory leak (a dict that fills up and is not really deleteable). I'd like to make a system where at some point (before Kube's TERM for memory usage), the service sends a signal to Kube to do a rolling update to up a new pod, wait for it to be ready (I know how to do that with Probes), and kill the old one that has reached the limit.
What is the best practice to do this?
I wanted to do it with health checker + rollingUpdate (maxUnavailable: 0 maxSurge: 1), where there must always be 1 pod running, and sending an "unhealthy" message with the Probe, but, unless I misconfigured something else, Kube kills the pod and recreates it without creating a second pod (so I have a downtime while the new pod is being created (~15s)).
Obviously I don't really want to have 2 replicas on all these services because it will cost me a lot of money. Is there another solution?
I was thinking of using the Kube API but wanted to ask you if there was a smarter solution.
