2

I am trying to use aws spot instances (m5.large in eu-west-2 region) with a maximum bid equal to the price of on demand instances. According to https://aws.amazon.com/ec2/spot/instance-advisor/ these instances should have a < 5% frequency of interruption, however, after launching 40 such instances this morning, I have found that within the hour 34 of them were evicted by aws ("instance-terminated-no-capacity" according to the spot requests page on the ec2 dashboard).

This eviction rate looks much too high compared to both amazon's own advisor and other users experiences. Does anybody know what could be causing this behaviour, if there is any better way to debug it or predict it, or if this is just what I should expect from spot instances?

Thank you!

Michele Valotti
  • 25
  • 3

2 Answers2

1

Actually, for m5.large instance in eu-west-2 region(Oregon) it's 5%-10% frequency of interruption, so you can expect a max of 10%. I'm not saying that issue you are facing is because of this.

AWS terminates your spot instances because of any of these reasons,

  1. The Spot price is above the maximum price.
  2. There isn't enough capacity.
  3. Amazon EC2 can't meet the constraints you placed on your Spot request.

In your case, since you are seeing instance-terminated-no-capacity message it is definitely because of the second reason. Since you've asked for 40 such instances, the amazon spot instance pool might not have enough capacity at that time.

The capacity of available spot instances pool depends on the demand for regular instances, and when users ask for regular on-demand instances, AWS will start terminating spot instances to fulfil those requests if there is not enough capacity

Sam
  • 4,046
  • 8
  • 31
  • 47
  • Thank you for your reply, do you know if there is any way to know or predict how busy the instance pool is at a certain time? It just seems weird to me that with such a low frequency of interruption I am getting 90% of my instances evicted in the first hour – Michele Valotti Aug 31 '21 at 11:06
  • @MicheleValotti I'm afraid there is no way to predict the spot instance capacity pool as far as I know. The capacity of available spot instances depends on the demand for regular instances, and when users ask for regular on-demand instances, AWS will start terminating spot instances to fulfil those requests if there is not enough capacity. But you can do analysis by checking 'Pricing History' of spot instance and you can get an insight of when it's in high demand but still, it's all based on price. – Sam Aug 31 '21 at 12:49
  • 1
    thank you again, looking at the price history it seems for my kind of instance in my region the price has remained very stable over the last three months (fluctuating by a maximum of +/- 5%) but my experience with evictions has varied a lot more, some instances remaining up for 1 or 2 days (even though never for longer than that) and some like today getting terminated en masse after a few minutes. It seems strange that this variability would not be reflected in price (or somewhere else at least!) – Michele Valotti Aug 31 '21 at 13:39
1

In my experience spot interruption is highly variable, and for some instances more or less likely at different times of the day.

If you need 40 instances and they do not need to be in the same availability zone (AZ) to each other you might reduce the chance of a mass interruption of all/most instances if you spread the machines across different AZs within the region as each availability zone has its own pool of machines. Although you will likely increase the chance that some machines will be interrupted.

Note this is not an option if you are using EMR, then they have to be in the same AZ.

Andrew
  • 43
  • 5