Helm stable/nginx-ingress with autoscaling, HPA couldn't read CPU or memory metrics

Question

I install the latest version (1.24.5) of stable/nginx-ingress with autoscaling: true on a GKE cluster.

After finish deploying, HPA said that unable to read all metrics.

Can you help me to debug and fix it? If there is any more detail/log do you need, please kindly ask and I will provide it!

Thank you

Update:

I can run other HPA perfectly
kubectl get hpa returns

unknown/100m for CPU

unknown/100Mi for RAM

score 1 · Answer 1 · edited Mar 05 '20 at 18:19

1

It seems that since the HPA that is created by enabling the autoscaler on the ingress controller, it will apply the HPA on the label app=nginx-ingress. The nginx-ingress default-backend also has the same label, and thus also requires us to define the CPU/memory resource requests/limits as well.

You can find more info in this GitHub issue

edited Mar 05 '20 at 18:19

Toby Speight

27,591
48
66
103

answered Mar 05 '20 at 17:54

Jackie Luc

109
6

score 0 · Answer 2 · answered Nov 21 '19 at 17:58

0

I think that it could be that some of the pod’s containers do not have the relevant resource request set. Accordingly to the Kubernetes HPA documentation, when this happens the utilization for the pod can not be defined (and the autoscaler will not take any action for that metric).

You could do a quick check by running kubectl get hpa to list the current autoscalers. Or you could get a more detailed description with kubectl describe hpa. If you see some targets listed as <unknown> then it's most likely that the resource request was not set properly.

In case you have already set resources requests in for all your pods and you still see this message, maybe it would be helpful if you share more details about how you're defining requests and/or limits.

These articles from Kubernetes documentation could be useful:

Specify a memory request and memory limit

Specify a CPU request and a CPU limit

Regards!

answered Nov 21 '19 at 17:58

Judith Guzman

415
3
11

1

Here is my request for resource: ``` resources: limits: cpu: 100m memory: 64Mi requests: cpu: 100m memory: 64Mi ``` – Quoc-Anh Nguyen Nov 22 '19 at 01:43
In the meantime, I can have other HPAs run normally but only HPA for nginx has that problem – Quoc-Anh Nguyen Nov 22 '19 at 01:45
@AnhNguyen What's the output when you run `kubectl get hpa`? – Judith Guzman Nov 22 '19 at 17:42
1

It said `/100m` for CPU and `/100Mi` for RAM – Quoc-Anh Nguyen Nov 23 '19 at 03:32

score 0 · Answer 3 · answered Apr 18 '20 at 08:06

0

I cannot comment yet. As marandall said in the GitHub issue Jackie linked above, this can be fixed by define resources limit and add --set controller.useComponentLabel=true --set defaultBackend.useComponentLabel=true to helm command.

answered Apr 18 '20 at 08:06

Zhuoqing Wu

11
1

Helm stable/nginx-ingress with autoscaling, HPA couldn't read CPU or memory metrics

3 Answers3