0

Taking reference to this answer, in my case, I have an MIG with an autoscaler set with:

  • minNumReplicas: 0
  • a custom, per-group Cloud Monitoring metrics that is generated from an external application outside of the MIG instances.

Theoretically, this should allow me to scale to 0. However, the targetSize and recommendedSize of my MIG always stays at 1, even when my metric is definitely below the utilization target set. (I have tested that the scaling up and down between 1 and my maxNumReplicas work)

I even tried to force a resize to 0, using gcloud compute instance-groups managed resize my-mig --size=0. This initially shows me a targetSize of 0, while the autoscaler.recommendedSize continues to remain at 1. A few seconds later, targetSize bounces back to 1, and I have this annoying instance that just won't go away.

Is this behaviour expected? How can I force recommendedSize to 0 (or essentially how can I make the MIG to go down to 0, which is the minNumReplicas that I have set)?

Erick Melgoza
  • 11
  • 3
chaooder
  • 1,358
  • 1
  • 17
  • 37

2 Answers2

0

Regarding the issue where you can not apply scale down to 0 even after a forced resize in MIG, when you want to maintain a metric at a target value, specify a utilization target. The autoscaler creates VMs when the metric value is above the target and deletes VMs when the metric value is below the target. When you use such metrics that originate from a VM, the MIG cannot scale to 0, because the autoscaler requires at least one VM to publish a metric value. For this reason, you can not apply scale down to 0. Here you can read more about scale based monitoring metrics.

A requirement for creating a custom metric, just like your set up, is: If the autoscaling configuration uses data from each VM in the group, set up instances in your MIG so that each VM exports the custom metric. The exported values from each VM must be associated with a gce_instance monitored resource. This means custom metrics come from a single instance inside the MIG, which explains why you cannot downscale to 0. You cannot change this behavior.

Horacio Garcia Gonzalez
  • 1
  • 1
  • I'm currently using a custom metric (independent of VM metric), and have already specified a utilization target (gauge). The current metric value is always < utilization target value, and can be verified from Monitoring, but there's always still 1 VM lingering around – chaooder Sep 02 '22 at 02:22
  • 1
    Still 1 VM lingering around happens due to the autoscaler gradually adding or removing VMs at a rate that is proportional to how much the metric is above or below the target. In other words the scale based on [custom metrics](https://cloud.google.com/compute/docs/autoscaler/scaling-cloud-monitoring-metrics#autoscale_based_on_a_custom_metric) and based on each VM in a MIG , always the autoscaler requires at least one VM to publish a metric value and cannot scale to 0. – Horacio Garcia Gonzalez Sep 05 '22 at 22:18
  • So you're saying, even though my custom metric is published externally and its values can still be published even when there is 0 VM in the MIG, the MIG still cannot scale down to 0 VM? – chaooder Sep 06 '22 at 08:04
  • @chaooder Which autoscaling signals are you using in your custom metric? – Hector Martinez Rodriguez Sep 06 '22 at 17:38
  • @Hector What do you mean by autoscaling signals? I have configured autoscaling based on a [custom metric](https://cloud.google.com/compute/docs/autoscaler/scaling-cloud-monitoring-metrics#:~:text=The%20custom%20metric%20exports%20data%20for%20the%20global%20monitored%20resource%2C%20that%20is%2C%20it%20is%20not%20associated%20with%20any%20specific%20VM.) with a particular gauge utilization target. This custom metric is emitted by an external application outside of the MIG to cloud monitoring, with a global monitored resource type. – chaooder Sep 08 '22 at 06:02
  • @chaooder An autoscaling signal is the basic component of an [autoscaling policy](https://cloud.google.com/compute/docs/autoscaler#autoscaling_policy), you can autoscale managed instance groups based on these signals: CPU utilization, Serving capacity of an external HTTP(S) load balancer, Monitoring metrics, Schedules. Similar to what is mentioned [here](https://stackoverflow.com/a/70707797/16923081). Additionally, I recommend you to read [this document](https://cloud.google.com/compute/docs/autoscaler/multiple-signals). – Hector Martinez Rodriguez Sep 08 '22 at 14:52
  • @HectorMartinezRodriguez the links that you have shared are exactly what I have shared across this post also :) In short, the signal I'm using is a custom metric emitted by an external application outside of the MIG to cloud monitoring, with a global monitored resource type. This means signals are available even when there is 0 VMs in the MIG (that is if I can get them to 0, which I can't at the moment). – chaooder Sep 09 '22 at 03:30
  • @chaooder, Yes, I did reference the [same question](https://stackoverflow.com/a/70707797/16923081) but only to complement my comment above about autoscaling signals because the answer lists the available signals to use with [Cloud Monitoring metrics](https://cloud.google.com/compute/docs/autoscaler/scaling-cloud-monitoring-metrics), this includes custom metrics. _It's not an issue but an expected behavior, this autoscaling limitation for MIGs is per design, you cannot downscale to 0_. – Hector Martinez Rodriguez Sep 09 '22 at 17:30
  • Completely correct – Horacio Garcia Gonzalez Sep 09 '22 at 21:16
  • @HectorMartinezRodriguez according to the [question](https://stackoverflow.com/a/70707797/16923081) you posted, it mentions scaling to 0 is possible if 1) minNumReplicas is set to 0 2) Only these autoscaling signals are used: schedules or per-group Cloud Monitoring metrics (or both). In my case, I fulfill both criteria, and if the reference answer is correct I should be able to downscale to 0. Does this mean the reference answer is not correct and there's no way to scale to 0 for MIG? – chaooder Sep 11 '22 at 00:20
  • @chaooder A [requirement for creating a custom metric](https://cloud.google.com/compute/docs/autoscaler/scaling-cloud-monitoring-metrics#custom_metrics), just like your set up, states: If the autoscaling configuration uses data from each VM in the group, set up instances in your MIG so that each VM exports the custom metric. The exported values from each VM must be associated with a gce_instance monitored resource. This means custom metrics come from a single instance inside the MIG, which explains why you cannot downscale to 0. You cannot change this expected behavior. – Horacio Garcia Gonzalez Sep 12 '22 at 22:33
  • @HoracioGarciaGonzalez my autoscaling configuration *doesn't* use data from my VMs at all; as I have explained in the post as well as several comments above, it comes from an external application that publishes data to GCP monitoring as a custom metric, which can then be used as a *per-group* Cloud Monitoring metric. Hence, even if there are no VM instances, the metric will still be published continuously by my external application. – chaooder Sep 13 '22 at 05:42
  • This really boils down to 2 simple questions: 1) does MIG support autoscaling to 0 in any way (excluding scheduling)? 2) if it does, under what circumstances can an MIG autoscale to 0? – chaooder Sep 13 '22 at 05:49
  • @chaooder Let's change the approach here. _1) Does MIG support autoscaling to 0 in any way (excluding scheduling)_? Yes, an autoscaling policy where the only scaling signal is a custom, per-group metric should be able to scale in to 0. – Hector Martinez Rodriguez Sep 13 '22 at 16:49
  • _2) if it does, under what circumstances can an MIG autoscale to 0?_ Keep in mind that if the metric shows any utilization, autoscaler will not scale to 0 because Autoscaler cannot recommend 0.1 VMs, so if there is a need for 0.1 VMs, any fractional utilization will be rounded up to 1 VM. – Hector Martinez Rodriguez Sep 13 '22 at 16:50
  • As reference, [here](https://cloud.google.com/compute/docs/autoscaler/understanding-autoscaler-decisions) you can read _how the Autoscaler takes decisions_. This includes how it recommends the amount of VMs to scale in or out, the 10 minute stabilization period and gaps between target and actual utilization metrics. – Hector Martinez Rodriguez Sep 13 '22 at 22:03
  • @HectorMartinezRodriguez thanks, so based on this, I clearly fulfill both criteria: 1) I am using only 1 custom, per group metric as the scaling signal 2) my custom metric is consistently at 0 value (below my target of 0.5) so there shouldn't be fractional utilization. Why is it that there's always still 1 VM persisting in the MIG, even after waiting for hours (way above the 10min stabilization period)? – chaooder Sep 14 '22 at 03:31
  • In my opinion, the reason could be the [gap between your 0.5 target and the actual utilization metric](https://cloud.google.com/compute/docs/autoscaler/understanding-autoscaler-decisions#gaps_between_target_and_actual_utilization_metrics). But it's impossible to determine the exact root cause without inspecting your GCP project only using public documentation, please engage GCP Support [here](https://cloud.google.com/support-hub), file a support case so they can investigate and give you the exact root cause. – Hector Martinez Rodriguez Sep 14 '22 at 05:27
0

Try not to send any custom metric data at all to trigger downscale to 0, because any metric data that is received (even a metric with zero value) will keep the VM size to at least 1.

See my screenshot here

Hope it helps, thank you

Aldi Permana
  • 1
  • 1
  • As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Dec 30 '22 at 08:19