Parse an AWS CloudFormation template with the PyYAML library

Question

I am writing a custom Python application using the PyYAML library that needs to read in AWS CloudFormation YAML templates.

I know the templates are valid CloudFormation templates, because I tested them using validate-template:

▶ aws cloudformation validate-template --template-body file://cloudformation.yml

When I try to read them using the PyYAML library, however, I get errors like:

yaml.scanner.ScannerError: mapping values are not allowed here

and

could not determine a constructor for the tag "!Sub"

and others.

By way of example, I try this AWS example template:

▶ curl -s \
    https://raw.githubusercontent.com/awslabs/aws-cloudformation-templates/master/aws/services/CloudFormation/FindInMap_Inside_Sub.yaml \
    -o FindInMap_Inside_Sub.yaml

And then:

▶ python
Python 2.7.15 (default, Nov 27 2018, 21:40:55) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>> yaml.load(open('FindInMap_Inside_Sub.yaml'))

Which leads to:

yaml.constructor.ConstructorError: could not determine a constructor for the tag '!FindInMap'
  in "FindInMap_Inside_Sub.yaml", line 89, column 45

How can I parse a CloudFormation YAML file using a library like PyYAML or others?

score 21 · Answer 1 · edited Apr 16 '21 at 17:49

It is possible to use the cfn_tools library that ships with the aws-cfn-template-flip project.

Install the library:

▶ pip install cfn_flip

Then the simplest Python to read in the template might be:

#!/usr/bin/env python
  
import yaml
from cfn_tools import load_yaml, dump_yaml

text = open('./FindInMap_Inside_Sub.yaml').read()
data = load_yaml(text)

print(dump_yaml(data))

This library is not really documented but there are also various methods in there for customising the formatting of the output worth exploring.

score 6 · Answer 2 · answered Jun 20 '18 at 08:47

6

Their aws-cfn-template-flip project that converts cfn templates to/from json and yaml is a good starting point. Example check out the yaml_loader.py script. It shows how it's adding yaml constructors. At the bottom, you'll see:

CfnYamlLoader.add_constructor(TAG_MAP, construct_mapping)
CfnYamlLoader.add_multi_constructor("!", multi_constructor)

You'll probably be interested in the construct_mapping method there. From there, you can look how the code works.

answered Jun 20 '18 at 08:47

Thank you so much for these informations. I found another workaround for my specific concern, but they will probably help me in other automation issues. – nixmind Jun 22 '18 at 12:15
1

In fact my real matter was to find a solution for AWS cfn templates parameters overrides an serverless applications deployment (artifacts and packages) in AWS Service Catalog, through an infrastructure system I set up with aws Management and CI/CD tools. I definitively decided to separate infrastructure from configuration and parameters, by using AWS Systems Manager Parameter Store. But I'll use these tools you indicated in another stage I intend to write a full article on what I did and what issues encountered exactly... – nixmind Jun 25 '18 at 08:59
Where is that article ;) ? – Baptiste Pernet Apr 10 '20 at 00:20

score 4 · Answer 3 · answered Jun 17 '20 at 20:50

I had some trouble with Alex's answer, because it was automatically converting my CF template into long form. So any !Ref Thing call was being converted into a dictionary mapping.

If you want to match the original input of your input.template file, use this:

from cfn_tools import load_yaml
import cfn_flip.yaml_dumper
import yaml


with open('input.template') as f:
    raw = f.read()
    data_dict = load_yaml(raw)

with open('output.template', 'w') as f:
    dumper = cfn_flip.yaml_dumper.get_dumper(clean_up=False, long_form=False)
    raw = yaml.dump(
        data_dict,
        Dumper=dumper,
        default_flow_style=False,
        allow_unicode=True
    )
    f.write(raw)

You can also change clean_up=False to True to perform some smart formatting, which worked well in my case.

I found this after running the cfn cli tool and seeing the correct short form output on my template. Then I used that main file as a reference and followed the code path.

Thanks for this! Would you happen to know how to keep variables of constructors in the same line? For example, even after using this I'm still not getting the result I want: `- !FindInMap [PrivateLink, EndPoint, SubnetId1]`. Instead I'm getting `- !FindInMap` and then it drops a line and passes the rest of the variables `- PrivateLink - EndPoint - SubnetId1` — Daniel, Feb 17 '21 at 12:05
Those 2 things should be functionally equivalent @Daniel, in cloudformation yaml both of those ways are treated as lists. But no I don't know of a way. But I wouldn't bother investigating since it's just stylistic — CornSmith, Mar 05 '21 at 13:55

Parse an AWS CloudFormation template with the PyYAML library

3 Answers3

Linked