11

Now that AWS have a Pricing API, how could one use Boto3 to fetch the current hourly price for a given on-demand EC2 instance type (e.g. t2.micro), region (e.g. eu-west-1) and operating system (e.g. Linux)? I only want the price returned. Based on my understanding, having those four pieces of information should be enough to filter down to a singular result.

However, all the examples I've seen fetch huge lists of data from the API that would have to be post-processed in order to get what I want. I would like to filter the data on the API side, before it's being returned.

toringe
  • 1,194
  • 3
  • 12
  • 18
  • I was reading through As per https://github.com/boto/boto3/issues/799, they say only spot instance pricing is available. – Kush Vyas Aug 03 '18 at 14:17
  • 1
    That is correct for the EC2 API, but Boto3 now has a separate Pricing API that do return on-demand prices. – toringe Aug 04 '18 at 10:49

4 Answers4

35

Here is the solution I ended up with. Using Boto3's own Pricing API with a filter for the instance type, region and operating system. The API still returns a lot of information, so I needed to do a bit of post-processing.

import boto3
import json
from pkg_resources import resource_filename

# Search product filter. This will reduce the amount of data returned by the
# get_products function of the Pricing API
FLT = '[{{"Field": "tenancy", "Value": "shared", "Type": "TERM_MATCH"}},'\
      '{{"Field": "operatingSystem", "Value": "{o}", "Type": "TERM_MATCH"}},'\
      '{{"Field": "preInstalledSw", "Value": "NA", "Type": "TERM_MATCH"}},'\
      '{{"Field": "instanceType", "Value": "{t}", "Type": "TERM_MATCH"}},'\
      '{{"Field": "location", "Value": "{r}", "Type": "TERM_MATCH"}},'\
      '{{"Field": "capacitystatus", "Value": "Used", "Type": "TERM_MATCH"}}]'


# Get current AWS price for an on-demand instance
def get_price(region, instance, os):
    f = FLT.format(r=region, t=instance, o=os)
    data = client.get_products(ServiceCode='AmazonEC2', Filters=json.loads(f))
    od = json.loads(data['PriceList'][0])['terms']['OnDemand']
    id1 = list(od)[0]
    id2 = list(od[id1]['priceDimensions'])[0]
    return od[id1]['priceDimensions'][id2]['pricePerUnit']['USD']

# Translate region code to region name. Even though the API data contains
# regionCode field, it will not return accurate data. However using the location
# field will, but then we need to translate the region code into a region name.
# You could skip this by using the region names in your code directly, but most
# other APIs are using the region code.
def get_region_name(region_code):
    default_region = 'US East (N. Virginia)'
    endpoint_file = resource_filename('botocore', 'data/endpoints.json')
    try:
        with open(endpoint_file, 'r') as f:
            data = json.load(f)
        # Botocore is using Europe while Pricing API using EU...sigh...
        return data['partitions'][0]['regions'][region_code]['description'].replace('Europe', 'EU')
    except IOError:
        return default_region


# Use AWS Pricing API through Boto3
# API only has us-east-1 and ap-south-1 as valid endpoints.
# It doesn't have any impact on your selected region for your instance.
client = boto3.client('pricing', region_name='us-east-1')

# Get current price for a given instance, region and os
price = get_price(get_region_name('eu-west-1'), 't3.micro', 'Linux')
print(price)

This example outputs 0.0114000000 (hourly price in USD) fairly quickly. (This number was verified to match the current value listed here at the date of this writing)

toringe
  • 1,194
  • 3
  • 12
  • 18
  • As a FYI, I also had to create a mapping function to translate a region name (short id) to a full region name, as I usually use the short naming convention. I implemented this by utilizing the `endpoints.json` file included with the botocore module, as mentioned [here](https://github.com/boto/boto3/issues/1411). – toringe Sep 28 '18 at 12:04
  • Did you refine this any further? The solution above doesn't handle pagination, which is required if the initial filtering isn't specific enough. Are there other filters you've been successful with? – Brian Apr 02 '19 at 19:09
  • 1
    For what it is worth, I found this: https://aws.amazon.com/blogs/aws/aws-price-list-api-update-new-query-and-metadata-functions/ – Brian Apr 02 '19 at 19:38
  • @Brian can you elaborate on what you mean if the filtering isn't specific enough. I'm using the above code in production and haven't had an issue with pagination. (I now added the mapper function I mentioned in my comment). – toringe Apr 02 '19 at 20:02
  • Your use case might be specific enough actually since you are requesting an instanceType. If you leave that filter out, you get far more results that forces pagination, unless I'm doing something wrong. In my case, I'm looking for pricing, as well as just getting a list of instanceType. – Brian Apr 02 '19 at 20:07
  • That is probably true. But in my use-case I always have a specific instance type in addition to the other two facts when querying for a price. – toringe Apr 02 '19 at 20:19
  • 3
    It's probably a recent change in the aws api, but now you should add `'{{"Field": "capacitystatus", "Value": "Used", "Type": "TERM_MATCH"}},'\` to the filters to get the price for ondemand instances, otherwise code above returns unreliable results, because each price list contains 3 items in a random order per instance with different capacitystatus and different prices more info https://stackoverflow.com/a/55131304/1952982 – userqwerty1 May 07 '19 at 19:01
  • @toringe Could you update your answer with what userquerty1 said? In the current state, it returns 0 for m5d.8xlarge. – Caesar Dec 06 '19 at 05:48
  • 1
    Thanks userqwerty1 and Caesar for your suggestions. Answer has now been updated. – toringe Dec 16 '19 at 09:51
  • What are valid values for `location`? Is it something like `Oregon`, or something like `us-west-2`? – étale-cohomology Feb 02 '22 at 13:16
  • @étale-cohomology: The `location` field in the Pricing API contains region names like `US East (N. Virginia)` and `EU (Ireland)` instead of region codes like `us-east-1` and `eu-west-1`. Since most other APIs use region codes instead of names, I found it necessary to do a translation. – toringe Feb 14 '22 at 22:46
3

If you don't like the native function, then look at Lyft's awspricing library for Python. Here's an example:

import awspricing

ec2_offer = awspricing.offer('AmazonEC2')

p = ec2_offer.ondemand_hourly(
  't2.micro',
  operating_system='Linux',
  region='eu-west-1'
)

print(p) # 0.0126

I'd recommend enabling caching (see AWSPRICING_USE_CACHE) otherwise it will be slow.

jarmod
  • 71,565
  • 16
  • 115
  • 122
  • I've looked that Lyft's libarary, but the `awspricing.offer()` downloads the whole thing. Yes they do have local caching options, but that will not work in my environment (getting the price from a AWS Lambda function). The closest thing I've gotten is using the get-products method from the pricing api in boto3 with a filter. That is faster, but still returns a lot of data. – toringe Aug 03 '18 at 14:00
  • 2
    Then I suspect you'll have to roll your own solution. If you have a well-bounded set of results that you need to look up (e.g. just on-demand EC2 in 3 regions) then you could write a scheduled task that uses the awspricing library to cache the data, get the specific results that you expect to need, then store those to DynamoDB. Your Lambda price lookup would then simply query DynamoDB table. Or you could dump the entire pricing data into Elasticsearch, but that would be much more expensive to maintain than a small DynamoDB table. – jarmod Aug 03 '18 at 14:08
  • tnx, for the reply and suggestions. – toringe Aug 03 '18 at 14:16
2

I have updated toringe's solution a bit to handle different key errors

def price_information(self, instance_type, os, region):
        # Search product filter
        FLT = '[{{"Field": "operatingSystem", "Value": "{o}", "Type": "TERM_MATCH"}},' \
              '{{"Field": "instanceType", "Value": "{t}", "Type": "TERM_MATCH"}}]'
    
        f = FLT.format(t=instance_type, o=os)
        try:
            data = self.pricing_client.get_products(ServiceCode='AmazonEC2', Filters=json.loads(f))
            instance_price = 0
            for price in data['PriceList']:
                try:
                    first_id =  list(eval(price)['terms']['OnDemand'].keys())[0]
                    price_data = eval(price)['terms']['OnDemand'][first_id]
                    second_id = list(price_data['priceDimensions'].keys())[0]
                    instance_price = price_data['priceDimensions'][second_id]['pricePerUnit']['USD']
                    if float(price) > 0:
                        break
                except Exception as e:
                    print(e)
            print(instance_price)
            return instance_price
        except Exception as e:
            print(e)
            return 0
Dev-2019
  • 547
  • 3
  • 11
0

Based on other answers, here's some code that returns the On Demand prices for all instance types (or for a given instance type, if you add the search filter), gets some relevant attributes for each instance type, and pretty-prints the data.

It assumes pricing is the AWS Pricing client.

import json

def ec2_get_ondemand_prices(Filters):
  data  = []
  reply = pricing.get_products(ServiceCode='AmazonEC2', Filters=Filters, MaxResults=100)
  data.extend([json.loads(r) for r in reply['PriceList']])
  while 'NextToken' in reply.keys():
    reply = pricing.get_products(ServiceCode='AmazonEC2', Filters=Filters, MaxResults=100, NextToken=reply['NextToken'])
    data.extend([json.loads(r) for r in reply['PriceList']])
    print(f"\x1b[33mGET \x1b[0m{len(reply['PriceList']):3} \x1b[94m{len(data):4}\x1b[0m")

  instances = {}
  for d in data:
    attr = d['product']['attributes']
    type = attr['instanceType']
    if type in data:  continue

    region  = attr.get('location',              '')
    clock   = attr.get('clockSpeed',            '')
    type    = attr.get('instanceType',          '')
    market  = attr.get('marketoption',          '')
    ram     = attr.get('memory',                '')
    os      = attr.get('operatingSystem',       '')
    arch    = attr.get('processorArchitecture', '')
    region  = attr.get('regionCode',            '')
    storage = attr.get('storage',               '')
    tenancy = attr.get('tenancy',               '')
    usage   = attr.get('usagetype',             '')
    vcpu    = attr.get('vcpu',                  '')

    terms    = d['terms']
    ondemand = terms['OnDemand']

    ins      = ondemand[next(iter(ondemand))]
    pricedim = ins['priceDimensions']
    price    = pricedim[next(iter(pricedim))]
    desc     = price['description']
    p        = float(price['pricePerUnit']['USD'])
    unit     = price['unit'].lower()

    if 'GiB' not in ram: print('\x1b[31mWARN\x1b[0m')
    if 'hrs'!=unit:      print('\x1b[31mWARN\x1b[0m')

    if p==0.: continue
    instances[type] = {'type':type, 'market':market, 'vcpu':vcpu, 'ram':float(ram.replace('GiB','')), 'ondm':p, 'unit':unit, 'terms':list(terms.keys()), 'desc':desc}

  instances = {k:v for k,v in sorted(instances.items(), key=lambda e: e[1]['ondm'])}
  for ins in instances.values():
    p = ins['ondm']
    print(f"{ins['type']:32} {ins['market'].lower()}\x1b[91m: \x1b[0m{ins['vcpu']:3} vcores\x1b[91m, \x1b[0m{ins['ram']:7.1f} GB, \x1b[0m{p:7.4f} \x1b[95m$/h\x1b[0m, \x1b[0m\x1b[0m{p*720:8,.1f} \x1b[95m$/m\x1b[0m, \x1b[0m\x1b[0m{p*720*12:7,.0f} \x1b[95m$/y\x1b[0m, \x1b[0m{ins['unit']}\x1b[91m, \x1b[0m{ins['terms']}\x1b[0m")
    # print(desc, , sep='\n')

  print(f'\x1b[92m{len(instances)}\x1b[0m')

flt = [
  # {'Field': 'instanceType',    'Value': 't4g.nano',  'Type': 'TERM_MATCH'},  # enable this filter to select only 1 instance type
  {'Field': 'regionCode',      'Value': 'us-east-2', 'Type': 'TERM_MATCH'},  # alternative notation?: {'Field': 'location', 'Value': 'US East (Ohio)', 'Type': 'TERM_MATCH'},
  {'Field': 'operatingSystem', 'Value': 'Linux',     'Type': 'TERM_MATCH'},
  {'Field': 'tenancy',         'Value': 'shared',    'Type': 'TERM_MATCH'},
  {'Field': 'capacitystatus',  'Value': 'Used',      'Type': 'TERM_MATCH'},
]
ec2_get_ondemand_prices(Filters=flt)
étale-cohomology
  • 2,098
  • 2
  • 28
  • 33