5

Problem setting. Suppose I have 2 pods, A and B. I want to be able to dynamically scale pod A based on some arbitrary number from some arbitrary source. Suppose that pod B is such a source: for example, it can have an HTTP server with an endpoint which responds with the number of desired replicas of pod A at the moment of request. Or maybe it is an ES server or a SQL DB (does not matter).

Question. What kubernetes objects do I need to define to achieve this (apart from HPA)? What configuration should HPA have to know that it needs to look up B for current metric? How should API of B look like (or is there any constraints?)?

Research I have made. Unfortunately, the official documentation does not say much about it, apart from declaring that there is such a possibility. There are also two repositories, one with some go boilerplate code that I have trouble building and another one that has no usage instructions whatsoever (though allegedly does fulfil the "external metrics over HTTP" requirement).

By having a look at the .yaml configs in those repositories, I have reached a conclusion that apart from Deployment and Service one needs to define an APIService object that registers the external or custom metric in the kubernetes API and links it with a normal service (where you would have your pod) and a handful of ClusterRole and ClusterRoleBinding objects. But there is no explanation about it. Also I could not even list existing APIServices with kubectl in my local cluster (of 1.15 version) like other objects.

Jonas
  • 121,568
  • 97
  • 310
  • 388
Timur Nugmanov
  • 803
  • 9
  • 16

1 Answers1

5

The easiest way will be to feed metrics into Prometheus (which is a commonly solved problem), and then setup a Prometheus-based HPA (also a commonly solved problem).

1. Feed own metrics to Prometheus

  • Start with Prometheus-Operator to get the cluster itself monitored, and get access to ServiceMonitor objects. ServiceMonitors are pointers to services in the cluster. They let your pod's /metrics endpoint be discovered and scraped by a prometheus server.
  • Write a pod that reads metrics from your 3rd party API and shows them in own /metrics endpoint. This will be the adapter between your API and Prometheus format. There are clients of course: https://github.com/prometheus/client_python#exporting
  • Write a Service of type ClusterIP that represents your pod.
  • Write a ServiceMonitor that points to a service.
  • Query your custom metrics thru Prometheus dashboard to ensure this stage is done.

2. Setup Prometheus-based HPA

This looks like a huge pile of work to get the HPA. However, only the adapter pod is a custom part here. Everything else is a standard stack setup in most of the clusters, and you will get many other use cases for it anyways.

Max Lobur
  • 5,662
  • 22
  • 35
  • Thank you for the effort of writing this down. However, while this answer provides a working workaround, this is not exactly what I am looking for. My question is about understanding the hard part of building this myself from ground up in Kubernetes. Nonetheless, thanks again – Timur Nugmanov Sep 03 '20 at 08:53
  • Note that if you want a service that responds with the number of desired replicas you need a custom Controller (or Operator), not the custom Metrics server. In HPA notation I don't see a way to query metric and put the resulting value directly to desired replicas. – Max Lobur Sep 03 '20 at 11:28
  • Look at the https://github.com/operator-framework/operator-sdk. The operator instance can take num replicas from the outside and control accordingly. This is basically the same as writing a pod that can modify other deployments, without HPA. – Max Lobur Sep 03 '20 at 11:31