8

This question spun out of this one. Now that I have a better understanding of what was going wrong there, and a workable, if imperfect, solution, I'm submitting a more focused follow-up (I'm still something of a novice at StackOverflow - please let me know if this contravenes etiquette, and I should follow-up on the original).

This page suggests that "You use AWS CodeBuild to build, locally test, and package your serverless application". However, when I include a sam build command in my buildspec.yml, I get the following log output, suggesting that sam is not installed on CodeBuild images:

[Container] 2018/12/31 11:41:49 Running command sam build --use-container 
sh: 1: sam: not found 

[Container] 2018/12/31 11:41:49 Command did not exit successfully sam build --use-container exit status 127 
[Container] 2018/12/31 11:41:49 Phase complete: BUILD Success: false 
[Container] 2018/12/31 11:41:49 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: sam build --use-container. Reason: exit status 127

Furthermore, if I install SAM with pip install aws-sam-cli, running sam build --use-container in CodeBuild gives an error. sam build itself succeeds, but since it doesn't install test dependencies, I'd still need to use pip install -r tests/requirements-test.txt -t . to be able to run tests in CodeBuild. Moreover, this suggests that --use-container is required for "packages that have natively compiled programs").

This makes me wonder whether I'm trying to do something wrong. What's the recommended way of building SAM services in a CI/CD workflow on AWS?

scubbo
  • 4,969
  • 7
  • 40
  • 71

4 Answers4

6

2019_10_18 - Update (confirming @Spiff answer above):

Apparently Codebuild now work seamlessly with SAM, that's all I needed in buildspec.yml for a lambda using pandas and psycopg2-binary:

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.7

  pre_build:
    commands:
      - python -m unittest discover tests

  build:
    commands:
      - sam build

  post_build:
    commands:
      - sam package --output-template-file packaged.yaml --s3-bucket my-code-pipeline-bucketz

artifacts:
  type: zip
  files:
    - packaged.yaml

Cheers

saza
  • 460
  • 6
  • 7
4

Please see below for buildspec.yaml that works for me when using AWS SAM with AWS CodeBuild, with cloudformation.yml

phases:
  build:
    commands:
      - pip install --user aws-sam-cli
      - USER_BASE_PATH=$(python -m site --user-base)
      - export PATH=$PATH:$USER_BASE_PATH/bin
      - sam build -t cloudformation.yml
      - aws cloudformation package --template-file .aws-sam/build/template.yaml --s3-bucket <TARGET_S3_BUCKET> --output-template-file cloudformation-packaged.yaml
      - aws s3 cp ./cloudformation-packaged.yaml <TARGET_S3_BUCKET>/cloudformation-packaged.yaml

In the result I get a deployment package and packaged cloudformation template in the TARGET_S3_BUCKET.

For each function in the ./src folder, I have a requirements.txt file that includes all the dependencies, but I dont run pip install -r requirements.txt manually.

Lech Migdal
  • 3,828
  • 5
  • 36
  • 63
  • 1
    Hey Lech! Thanks for the example. Given that this presumably only installs your source requirements (not Test Requirements - like `requests-mock`, `pytest`, etc.), how do you run your tests during build? Does this setup allow you to use "packages that have natively compiled programs" (which usually required `sam build --use-container`)? Is there a reason why you're exporting the `cloudformation-packaged.yaml` file to S3, rather than just referencing it directly in a "Deploy" phase of your CodePipeline? – scubbo Dec 31 '18 at 13:26
  • Hi, all good questions! :) 1. I dont do tests here, I only use CodeBuild for building. If you'd like to do tests, yes, I'd say you should run pip. 2. As for moving packaged file, potentially I guess you could do this with CodeDeploy, but since I have very specific and unusual structure of folders, I ended up doing it manually with AWS CLI :) – Lech Migdal Dec 31 '18 at 13:34
  • Got it, thanks! I'll wait to accept a more complete and/or generally applicable answer, then - but it was helpful to see someone else's setup, thank you! – scubbo Dec 31 '18 at 13:53
  • @LechMigdal just curious, was there a reason you didn't use the `sam package` command instead of the cfn package/s3 copy commands? – abbottdev Feb 25 '19 at 17:06
  • Greetings! It worked great for me but I'm wondering. Is there any way to avoid installing AWS SAM CLI each time the build step runs? It seems pretty inefficient. – Tommy Aug 01 '19 at 15:42
  • Hey @LechMigdal any reason why you wouldn't use `sam package` command for your last steps? – Stefano Messina Sep 04 '19 at 08:47
2

If you want to run sam build command in CodeBuild, you must install aws-sam-cli first (probably in the install phase of buildspec.yml file) i.e. by running pip install aws-sam-cli command or alike.

--use-container option in the sam build command will cause the command to pull in the Docker image resembling the AWS Lambda execution environment, then run the container from this Docker image to pip install (if your lambda is written in Python) your function dependencies for creating your lambda deployment package. This will ensure that the lambda function will use native compiled libraries that are compatible with the actual runtime environment of AWS Lambda.

Therefore, if you specify --use-container option for sam build command running in CodeBuild, you also need to make sure that a Docker image used by your CodeBuild build project must support Docker runtime.

The most easiest way is to use CodeBuild build environment named aws/codebuild/standard:2.0 Docker image. Enabling Docker runtime in runtime-versions property in the install phases of your buildspec.yml. Also you might need to enable PrevilegedMode of your CodeBuild project in order to connect with Docker daemon from your build environment.

onelaview
  • 1,121
  • 15
  • 17
0

As of October 2019 I had no issues whatsoever deploying a serverless application with codebuild using sam build,

First of all --user is not needed for pip install aws-sam-cli. In fact including --user appears to be the only reason that sam is not in the path.

In addition the --use-container is not needed either as long as no native libraries are built, like psycopg

Spiff
  • 3,873
  • 4
  • 25
  • 50