aws cloudformation package transforms some local paths to S3 uris but not all

Question

Summary:

I use local paths to reference code for lambda functions and a state machine in the template.yml file describing my cloudformation setup. Transforming these to S3 uris with aws cloudformation package works for the lambda functions, but not the state machine that I'm trying to add to the setup.

Details:

I have a SAM/Cloudformation template, template.yml, that relies on paths that apply to my local repo to access both lambda functions and state machine setup files.

template.yml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31' # specifies that this is a SAM template, will be transformed to CFN on build

Resources:
  WriteDataToDb:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src                                           <- here
      Handler: writer.write_to_db
...
  ProcessDataStateMachine:
    Type: AWS::Serverless::StateMachine
    Properties:
      DefinitionSubstitutions: 
        WriteDataToDbFunctionArn: !GetAtt WriteDataToDb.Arn 
      DefinitionUri: statemachine/dataprocessor.json         <- here
...

I use the aws cloudformation package command to translate my template.yml file into a pure cloudformation file, packaged.yml, and to store the relevant code in a S3 bucket, deploymentpackages (dummy name). The references to the lambda functions in packaged.yml have correctly been translated into S3 uris. The uri for the state machine file however, is not converted to an S3 uri but remains as the same local path.

packaged.yml:

  ...
  WriteDataToDb:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: s3://deploymentpackages/33fc42027aae846c97ca8e13fec1bba7   <- translated
      Handler: writer.write_to_db
  ...
  ProcessDataStateMachine:
    Type: AWS::Serverless::StateMachine
    Properties:
      DefinitionSubstitutions:
        WriteDataToDbFunctionArn:
          Fn::GetAtt:
          - WriteDataToDb
          - Arn
      DefinitionUri: statemachine/dataprocessor.json                  <- not translated

Then when I try to create a change set from my packaged.yml file, I get the error 'DefinitionUri' is not a valid S3 Uri of the form 's3://bucket/key', which of course makes sense since it isn't.

My repo is organized in the following way:

├── src
│   ├── __init__.py
│   ├── ...
│   └── writer.py
├── statemachine
│   └── dataprocessor.json
├── template.yml

and I have verified that both the src and statemachine folders make it to the deploymentpackages bucket.

Why would the aws cloudformation package command work for the lambda uri but not the state machine one?

Good question, I'm using Codebuild with a docker image aws/codebuild/standard:2.0-1.9.0 which is something I set up last year and could be outdated. I'm having trouble finding the exact cli version number though. — Nanna, Sep 17 '20 at 16:21
I suspect its happening due to cli version < 0.52.0 An issue is already `closed` for the same [here](https://github.com/aws/aws-sam-cli/issues/2027) on GitHub. Updating the CLI should solve the problem. — CK__, Sep 17 '20 at 16:39
To get the cli version, you could run a command in `aws --version` in your `buildspec.yml` file(preferably in `pre_build` phase). — CK__, Sep 17 '20 at 16:43
Thanks for the advice, unfortunately that doesn't seem to be the culprit, I get: `aws-cli/1.16.142 Python/3.6.7 Linux/4.14.186-110.268.amzn1.x86_64 exec-env/AWS_ECS_EC2 botocore/1.12.132`. This is after I updated my CodeBuild project to use `aws/codebuild/standard:3.0`, but the same error still comes up. — Nanna, Sep 18 '20 at 09:05

score 2 · Answer 1 · answered Sep 17 '20 at 22:42

Sadly, DefinitionUri for AWS::Serverless::StateMachine is not supported for such substitutions. In contrast CodeUri is supported, thus its correctly changed.

The supported properties and resources are:

    BodyS3Location property for the AWS::ApiGateway::RestApi resource
    Code property for the AWS::Lambda::Function resource
    CodeUri property for the AWS::Serverless::Function resource
    DefinitionS3Location property for the AWS::AppSync::GraphQLSchema resource
    RequestMappingTemplateS3Location property for the AWS::AppSync::Resolver resource
    ResponseMappingTemplateS3Location property for the AWS::AppSync::Resolver resource
    DefinitionUri property for the AWS::Serverless::Api resource
    Location parameter for the AWS::Include transform
    SourceBundle property for the AWS::ElasticBeanstalk::ApplicationVersion resource
    TemplateURL property for the AWS::CloudFormation::Stack resource
    Command.ScriptLocation property for the AWS::Glue::Job resource
    DefinitionS3Location property for the AWS::StepFunctions::StateMachine resource

Oh no! I was sure this was an option, since it is used in this tutorial: https://www.youtube.com/watch?v=YvxB9WRc2o8&ab_channel=FooBarServerless with this template.yml: https://github.com/mavi888/sam-step-functions-demo/blob/master/template.yml . Could the difference be that she uses `sam deploy` and I'm using `aws cloudformation package`? — Nanna, Sep 18 '20 at 09:13
@Nanna Not sure how sam works in this aspect. But sadly `aws cloudformation package` does not support it as listed in the answer. — Marcin, Sep 18 '20 at 10:13
You are right about the capabilities of `aws cloudformation package`, as it turns out I just needed `sam package`. Thanks for getting me a step closer to the solution. — Nanna, Sep 18 '20 at 12:17

score 1 · Accepted Answer · answered Sep 18 '20 at 12:15

I was under the impression when I asked this question that aws cloudformation package and sam package were equivalent, since aws cloudformation package was able to transform local lambda function paths into S3 uris without problems.

I am using Codebuild to handle the transformation, and here is the buildspec.yml file when I was having the state machine DefinitionUri error:

version: 0.2
phases:
  install:
    runtime-versions:
      python: 3.7
    commands:
      - aws cloudformation package --template-file template.yml --s3-bucket deploymentpackages --output-template-file packaged.yml --region eu-central-1
artifacts:
  files:
    - packaged.yml

As it turns out, I really did need to use sam package or sam deploy, so here is a buildspec.yml file that successfully transforms the DefinitionUri from a local path to an S3 uri, in case anyone else might hit the same wall:

version: 0.2
phases:
  install:
    runtime-versions:
      python: 3.7
    commands:
      - pip install --user aws-sam-cli
      - USER_BASE_PATH=$(python -m site --user-base)
      - export PATH=$PATH:$USER_BASE_PATH/bin
      - sam build -t template.yml
      - sam package --template-file .aws-sam/build/template.yaml --s3-bucket deploymentpackages --output-template-file packaged.yml --region eu-central-1
artifacts:
  files:
    - packaged.yml

Inspired by this answer.

aws cloudformation package transforms some local paths to S3 uris but not all

2 Answers2