I am trying to understand how to setup multiple python lambdas and a step function within one single serverless.yml with each python lambda having its own dependencies. All of my lambda functions collaborate in the context of a step function for a shared common goal. With this rationale, it makes sense to me to put all of the code under one serverless.yml file.
As part of my MANY hours of trial and error and reading I found about the serverless-python-requirements plugin for The Serverless Framework that helps in packaging python functions that rely on OS-specific python libraries and also allow the separation of multiple requirements.txt in case different lambdas require different dependencies.
So at this point my problem is that the generated package is not including the dependencies that I provide in the requirements.txt whenever each function has its own requirements.txt
These are my artifacts:
package.json
{
"engines": {
"node": ">=10.0.0",
"npm": ">=6.0.0"
},
"name": "example",
"version": "1.0.0",
"description": "",
"main": "index.js",
"dependencies": {
"serverless-python-requirements": "^5.1.0"
},
"devDependencies": {
"serverless": "^1.72.0"
},
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"license": "ISC"
}
serverless.yml
service: example
frameworkVersion: ">=1.72.0 <2.0.0"
plugins:
- serverless-python-requirements
custom:
stage: "${opt:stage, env:SLS_STAGE, 'local'}"
log_level: "${env:LOG_LEVEL, 'INFO'}"
pythonRequirements:
dockerizePip: true
provider:
name: aws
# profile: ${self:custom.profile}
stage: ${self:custom.stage}
runtime: python3.8
environment:
LOG_LEVEL: ${self:custom.log_level}
package:
individually: true
exclude:
- ./**
include:
- vendored/**
functions:
function1:
# module: folder1
handler: folder1/function1.handler
package:
include:
- 'folder1/**'
memorySize: 128
timeout: 60
function2:
# module: folder2
handler: folder2/function2.handler
package:
include:
- 'folder2/**'
memorySize: 128
timeout: 60
finally, my 2 python lambda functions are in separate folders and one of them requires specific dependencies:
- folder1
- function1.py
- requirements.txt
- folder2
- function1.py
function1.py
import json
import logging
import os
import sys
import pyjokes
log_level = os.environ.get('LOG_LEVEL', 'INFO')
logging.root.setLevel(logging.getLevelName(log_level))
_logger = logging.getLogger(__name__)
class HandlerBaseError(Exception):
'''Base error class'''
class ComponentIdentifierBaseError(HandlerBaseError):
'''Base Component Identifier Error'''
def handler(event, context):
'''Function entry'''
_logger.debug('Event received: {}'.format(json.dumps(event)))
body = {
"message": "Go Serverless v1.0! Your function executed successfully!",
"joke":pyjokes.get_joke()
}
resp = {
'status': 'OK',
"body": json.dumps(body)
}
_logger.debug('Response: {}'.format(json.dumps(resp)))
return resp
if __name__ == "__main__":
handler('', '')
requirements.txt
pyjokes==0.6.0
function2.py
import json
import logging
import os
import sys
log_level = os.environ.get('LOG_LEVEL', 'INFO')
logging.root.setLevel(logging.getLevelName(log_level))
_logger = logging.getLogger(__name__)
class HandlerBaseError(Exception):
'''Base error class'''
class ElasticSearchPopulatorBaseError(HandlerBaseError):
'''Base Component Identifier Error'''
def handler(event, context):
'''Function entry'''
_logger.debug('Event received: {}'.format(json.dumps(event)))
resp = {
'status': 'OK'
}
_logger.debug('Response: {}'.format(json.dumps(resp)))
return resp
Note: I did try using the module+handler keywords in the serverless.xml as recommended on this link: ttps://github.com/UnitedIncome/serverless-python-requirements without any success
Something that I noted is that if I use the module+handler as follows:
functions:
function1:
module: folder1
handler: function1.handler
package:
include:
- 'folder1/**'
memorySize: 128
timeout: 60
Then, when I try running the function locally using: serverless invoke local -f function1 --log I get an error saying:
ModuleNotFoundError: No module named 'function1'
Also, if anyone has an example of multiple lambdas with different requirements.txt that works I would be very gratelful, ideally something just different than the typical hello world examples, the hello worlds all work very well for me ;), but in scenarios like this one where I would like to setup common libraries, have different dependencies, etc, and use one common serverless.yml things seem to fall apart. Again, my opinion is that these lambdas will operate together under one step function umbrella so there's strong cohesion here and I think that their build and deployment should happen under one common serverless service.
