SAM build - does it also build layers?

Question

I'm new to both lambda's and also SAM - so if I've screwed anything simple up don't yell :D.

Summary: I can't get sam build to build a layer specified in template.yaml, it only builds the lambda function.

Background: I'm trying to build a lambda function in python3.7 that uses the skimage (scikit-image) module. To do that, I'm trying to use SAM to build and deploy it all. ...this is working

I'm trying to deploy the scikit-image module as a layer (and also build with SAM), rather than having it included in the lambda function direction ...this isn't working

As a start, I'm simply extending the standard SAM Hello World app.

I've got skimage working by simply add it to requirements.txt , then using sam build -u, then manually removing the numpy/scipy dependencies from the built package directory (I've got the AWS scipy/numpy layer included).

(I added import numpy, scipy.ndimage and skimage.draw to the standard hello world app, and included some test function calls to each)

requirements.txt:

requests
scikit-image

After that, everything works fine (running locally and/or on AWS).

However, I'd now like to move the skimage module out of my app and into a new custom layer (I'd like to have skimage in a layer to re-use for a few functions)

To set that up, I've created a dependencies directory and moved requirements.txt into there (leaving empty requirements.txt in the app directory). I then updated template.yaml to also specify the new layer:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-app

  Sample SAM Template for sam-app

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.7
      Layers:
              - arn:aws:lambda:us-west-2:420165488524:layer:AWSLambda-Python37-SciPy1x:2
              - !Ref SkimageLayer
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get

  SkimageLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: Skimage
      Description: Skimage module layer
      ContentUri: dependencies/
      CompatibleRuntimes:
              - python3.7
      RetentionPolicy: Retain
    DependsOn:
            - Skimage

directory structure:

▾ dependencies/                
    requirements.txt (responses and scikit-image)          
▸ events/                      
▾ hello_world/                 
    __init__.py                
    app.py                     
    requirements.txt (now empty)          
▸ tests/                       
  README.md                    
  template.yaml

However, when I run sam build -u with that template file, nothing gets built for the layer specified in ./dependencies: SkimageLayer in the template.yml file. However the HelloWorldFunction still gets built correctly (now of course without any included modules)

I'm working a bit based on this: https://noise.getoto.net/2019/02/06/working-with-aws-lambda-and-lambda-layers-in-aws-sam/. About halfway down he implies with how he does it that SAM build wont build a layer itself, as he manually runs npm install inside his node dependencies directory to install the node dependencies — Richard, Oct 15 '19 at 20:55

gustahrodrigues · Accepted Answer · 2020-06-12T11:52:10.077

Since SAM Cli version v0.50.0, it is building layers as part of sam build.

Design document could be a good starting point to understand how it works.

Basically you have to set a custom BuildMethod with your lambda's target runtime:

MyLayer:
Type: AWS::Serverless::LayerVersion
Properties:
  ContentUri: my_layer
  CompatibleRuntimes:
    - python3.8
Metadata:
  BuildMethod: python3.8 (or nodejs8.10 etc..)

Warning: For compiled language as Java, it has a issue which it tries to build layers before functions. It's expected to have it fixed on the next release (PR opened already).

score 2 · Answer 2 · answered Oct 16 '19 at 07:46

Quick answer - No, currently SAM does not build layers you define in a SAM template.yaml file.

It will only build any functions you define.

However (curiously) it will package (upload to S3) and deploy (setup within AWS, assign ARN so it can be used etc) any layers you define.

There is a feature request on the SAM github issues to implement layer building with SAM.

This can actually be hacked right now to get SAM to build your layers as well, by creating a dummy function in your SAM template file, as well as a layer entry, and having the layer ContentUri entry point to the .aws-sam build directory that gets created for the function.

See my post here on that.

That approach actually seems to work pretty well for twisting SAM right now to build your layers for you.

score 1 · Answer 3 · answered Jan 07 '20 at 21:05

1

I'm not sure if something changed recently but I'm able to do this without issue. My template file and structure is very similar to the OP except I've put all my common code into...

/dependencies/python/lib/python3.7/site-packages/

I didn't include a requirements.txt file in that directory... just the __init__.py file and various .py files that I need to import into my functions.

SAM then finds the code and builds the layer. You don't even need to zip the contents of the directory as some tutorials tell you to do.

The best part is Layers: is able to be put into the Globals: section of the template file so that the layer is available to all of your functions!

Globals:
  Function:
    Handler: main.lambda_handler
    Timeout: 10
    Runtime: python3.7
    Layers: 
        - !Ref HelperFunctions

Resources:
  HelperFunctions:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: MyHelperFunctions
      Description: My Lambda Layer with Helper Functions for accessing RDS, Logging, and other utilities.
      ContentUri: dependencies/
      CompatibleRuntimes:
        - python3.6
        - python3.7
      LicenseInfo: MIT
      RetentionPolicy: Delete

answered Jan 07 '20 at 21:05

Kristin Green

29
3

Appreciate this answer -- I'll give this approach a try. – John Lockwood Feb 08 '20 at 23:48
Problem with this right now is you build all the dependencies on your local. Some might not work after deploy. `numpy` for example would have this issue. – karuhanga Feb 11 '20 at 20:01
The path is important. The directory name of 'dependencies' is just one I chose but everything beneath it is important and must follow this configuration. If you do, the site packages should be found when uploading the layer. I haven't tried doing it with numpy but I fail to understand how it wouldn't work. – Kristin Green Feb 12 '20 at 21:05
1

@Karuhanga - when you use SAM, there is the option with sam build (I think it's -u flag?) to build inside an AWS lambda environment docker container. That way it builds locally, but uses the correct AWS environment. Therefore everything builds 'natively' for lambda. I use this without problems for numpy etc. – Richard Feb 19 '20 at 07:20
1

@Karuhanga - also NOTE - AWS supplies a pre-built numpy/scipy layer for you to use - you don't need to build your own layer for numpy/scipy if that's all you need a layer for. – Richard Feb 19 '20 at 07:22
@Richard how would you reference code built in a docker container in the contentUri? – karuhanga Apr 08 '20 at 22:32
In code, it's simply a bunch of packaged modules, so usual import statements. In SAM build, like this: AQSTestCTACR: Type: AWS::Serverless::Function Properties: FunctionName: aqs_test_ct_acr Runtime: python3.7 CodeUri: ./src/ Handler: ct_acr.lambda_handler MemorySize: 2048 Timeout: 15 Role: arn:aws:iam::XXXX Layers: - arn:aws:lambda:us-west-2:XXXX:layer:AWSLambda-Python37-SciPy1x:2 - arn:aws:lambda:us-west-2:XXXX:layer:AQSTestModules:3 – Richard Apr 10 '20 at 23:40
The above isn't formatted :(. But I build a lambda with reference to two layers - the one I built, and then also the 'included' AWS numpy/scipy layer – Richard Apr 10 '20 at 23:41
Thank you. Tried it. `sam` mounts your folder to the docker container. – karuhanga Jul 15 '20 at 16:28

score 1 · Answer 4 · answered Nov 16 '20 at 15:22

1

The AWS team must have made things easier, relative to these older answers. From the current docs, all you do is list a layer as a property in your template (Nov 2020):

ServerlessFunction:
  Type: AWS::Serverless::Function
  Properties:
    CodeUri: .
    Handler: my_handler
    Runtime: Python3.7
    Layers:
        - arn:aws:lambda:us-west-2:111111111111:layer:myLayer:1
        - arn:aws:lambda:us-west-2:111111111111:layer:mySecondLayer:1

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-layers.html

answered Nov 16 '20 at 15:22

Chase Freeman

11
1

The OP question is about building layers using SAM template and not re-using layer inside SAM template – VadimK Feb 10 '21 at 22:03
2

Great point @VadimK. Hopefully someone will still find use from my comment as it is related and there is not much out there. – Chase Freeman Feb 17 '21 at 14:12

Neil McGuigan · Answer 5 · 2020-02-18T01:08:43.233

I got it to work with the following script. Tested with Ubuntu 18 and CodeBuild

It pip install's the layer's requirements to .aws-sam/build/layername/python/. Then you can run sam package and sam deploy as normal

build-layers.py:

import yaml
import subprocess
import sys
import shutil

SAM_BUILD_PATH = ".aws-sam/build"

with open("template.yaml", "r") as f:
    template = yaml.safe_load(f)

for key, resource in template["Resources"].items():
    if resource["Type"] not in ["AWS::Serverless::LayerVersion", "AWS::Lambda::LayerVersion"]:
        continue
    properties = resource["Properties"]
    content_uri = properties["ContentUri"]
    layer_name = properties["LayerName"]
    requirements_path = f"{content_uri}/requirements.txt"

    subprocess.check_call([sys.executable, "-m", "pip", "install", "-r", requirements_path, "-t", f"{SAM_BUILD_PATH}/{layer_name}/python"])

shutil.copyfile("template.yaml", f"{SAM_BUILD_PATH}/template.yaml")

template.yaml:

Transform: AWS::Serverless-2016-10-31

Resources:
  pandas:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: pandas
      ContentUri: pandas
      CompatibleRuntimes:
        - python3.6
        - python3.7
        - python3.8
  sqlparse:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: sqlparse
      ContentUri: sqlparse
      CompatibleRuntimes:
        - python3.6
        - python3.7
        - python3.8

so call python build-layers.py first, then sam package then sam deploy

my directories look like this:

lambda
  layers
    pandas
      requirements.txt (content = pandas)
    sqlparse
      requirements.txt (content = sqlparse)
  template.yaml
  build-layers.py

buildspec.yml:

---  # build spec for AWS CodeBuild

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.8
    commands:
      - pip install aws-sam-cli
  build:
    commands:
      - cd lambda/layers
      - python build-layers.py
      - sam package --s3-bucket foo --s3-prefix sam/lambda/layers | sam deploy --capabilities CAPABILITY_IAM -t /dev/stdin --stack-name LAYERS

Hey Neil - I'll have to check that. ...you got it to work _without_ specifying a function? The only way previously to 'fool' SAM into building a layer for you was to associate it with a dummy function. You couldn't specify only a layer in the template yaml file. But you seem to be able to do that! That would be good if they've changed that now — Richard, Feb 19 '20 at 07:25

SAM build - does it also build layers?

5 Answers5