1

I would like to be able to specify multiple resources using a single entry in serverless.yml.

My use case is as follows:

reportFatalError:
    handler: "handler.reportFatalError"
    events:
    - cloudwatchLog:
        # "*" implies wish to insert wildcard
        logGroup: "/aws/lambda/*-${opt:stage}-*"
        filter: "\"FATAL\""

Naturally, I would like the Fatal Error Reporter to report the Fatal Error from any logGroup, and explicitly specifying all of them is a recipe for poorly maintained serverless.yml in the future.

Is there some way to specify a wildcard or loop over the logGroups as serverless deploys?

Jay
  • 43
  • 1
  • 6

2 Answers2

3

Unfortunately there isn't a way to do this with wildcards. This is a limitation of AWS, not Serverless.

You could write a script that loaded a list of all your CloudWatch logGroups and then applied those events to your reportFatalError function on deployment.

See here: https://serverless.com/framework/docs/providers/aws/guide/variables/#reference-variables-in-javascript-files

Jeremy Daly
  • 46
  • 1
1

I also faced that issue recently and after researching the topic I ultimately decided to go with the implementation leveraging JavaScript based on what @Jeremy suggested. Frankly, it seems there is no alternative apart from having a long, long serverless.yml configuration file. I'll be more than happy to learn about other options if there are any.

So as a follow-up, I decided to share a minimal working example in case someone else needs to provide the same capability in the project and wonders how to actually achieve that.

Minimal example

Project structure

.
├── _serverless
│   └── cloudwatch_log_events.js
├── handler.py
└── serverless.yml

_serverless/cloudwatch_log_events.js

The _serverless directory is meant to store all Serverless Framework-related configuration files.

module.exports = async ({ options, resolveVariable }) => {
  const stage = await resolveVariable('sls:stage');
  const logGroups = [
    `/aws/lambda/foo-${stage}`,
    `/aws/lambda/bar-${stage}`,
  ];
  const logGroupFilter = '{ $.level = "error" }';

  return logGroups.map(function (logGroup) {
    return {
      cloudwatchLog: {
          logGroup: logGroup,
          filter: logGroupFilter
      }
    }
  });
}

handler.py

def foobar(event, context):
    return {
        "foo": "bar"
    }

serverless.yml

service: FoobarService


frameworkVersion: '3'


provider:
  name: aws
  runtime: python3.8
  stage: ${opt:stage, 'local'}


functions:
  foobar:
    name: foobar-${self:provider.stage}
    handler: handler.foobar
    events:
      ${file(_serverless/cloudwatch_log_events.js)}

The output serverless.yml

Generated using the $ sls print command.

service: FoobarService
frameworkVersion: '3'
provider:
  name: aws
  runtime: python3.8
  stage: local
  region: us-east-1
  versionFunctions: true
functions:
  foobar:
    name: foobar-local
    handler: handler.foobar
    events:
      - cloudwatchLog:
          logGroup: /aws/lambda/foo-local
          filter: '{ $.level = "error" }'
      - cloudwatchLog:
          logGroup: /aws/lambda/bar-local
          filter: '{ $.level = "error" }'

Testing

Setup

The following tests involve Localstack, therefore it needs to be added as a dependency in the plugins. Updated config:

serverless.yml

service: FoobarService


frameworkVersion: '3'


provider:
  name: aws
  runtime: python3.8
  stage: ${opt:stage, 'local'}


plugins:
  - serverless-localstack


custom:
  defaultStage: local
  profile: default
  localstack:
    debug: true
    stages:
      - local
    host: http://localhost
    edgePort: 4566
    autostart: true
    lambda:
      mountCode: false
    docker:
      sudo: false


functions:
  # Note:
  # `foo` and `bar` functions were added here only for demo purposes, just to assign
  # existing CloudWatch groups as the `foobar` lambda's event source mappings.
  # Providing unexisting groups would result in a deployment failure.
  foo:
    name: foo-${self:provider.stage}
    handler: handler.foobar
  bar:
    name: bar-${self:provider.stage}
    handler: handler.foobar
  foobar:
    name: foobar-${self:provider.stage}
    handler: handler.foobar
    events:
      ${file(_serverless/cloudwatch_log_events.js)}

docker-compose.yml

It's a minimal Localstack config file needed for the demo. Just run it with $ docker-compose up.

version: '2.1'

services:
  localstack:
    container_name: "localstack"
    image: localstack/localstack:latest
    ports:
      - "4566-4599:4566-4599"
    environment:
      - SERVICES=iam,cloudformation,s3,lambda,sts
      - DEBUG=1
      - LAMBDA_EXECUTOR=local
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

Outcome

$ sls print --stage local
Running "serverless" from node_modules
config.options_stage: local
serverless.service.custom.stage: undefined
serverless.service.provider.stage: local
config.stage: local
Using serverless-localstack
Reconfiguring service acm to use http://localhost:4566
Reconfiguring service amplify to use http://localhost:4566
Reconfiguring service apigateway to use http://localhost:4566
Reconfiguring service apigatewayv2 to use http://localhost:4566
Reconfiguring service application-autoscaling to use http://localhost:4566
Reconfiguring service appsync to use http://localhost:4566
Reconfiguring service athena to use http://localhost:4566
Reconfiguring service autoscaling to use http://localhost:4566
Reconfiguring service batch to use http://localhost:4566
Reconfiguring service cloudformation to use http://localhost:4566
Reconfiguring service cloudfront to use http://localhost:4566
Reconfiguring service cloudsearch to use http://localhost:4566
Reconfiguring service cloudtrail to use http://localhost:4566
Reconfiguring service cloudwatch to use http://localhost:4566
Reconfiguring service cloudwatchlogs to use http://localhost:4566
Reconfiguring service codecommit to use http://localhost:4566
Reconfiguring service cognito-idp to use http://localhost:4566
Reconfiguring service cognito-identity to use http://localhost:4566
Reconfiguring service docdb to use http://localhost:4566
Reconfiguring service dynamodb to use http://localhost:4566
Reconfiguring service dynamodbstreams to use http://localhost:4566
Reconfiguring service ec2 to use http://localhost:4566
Reconfiguring service ecr to use http://localhost:4566
Reconfiguring service ecs to use http://localhost:4566
Reconfiguring service eks to use http://localhost:4566
Reconfiguring service elasticache to use http://localhost:4566
Reconfiguring service elasticbeanstalk to use http://localhost:4566
Reconfiguring service elb to use http://localhost:4566
Reconfiguring service elbv2 to use http://localhost:4566
Reconfiguring service emr to use http://localhost:4566
Reconfiguring service es to use http://localhost:4566
Reconfiguring service events to use http://localhost:4566
Reconfiguring service firehose to use http://localhost:4566
Reconfiguring service glacier to use http://localhost:4566
Reconfiguring service glue to use http://localhost:4566
Reconfiguring service iam to use http://localhost:4566
Reconfiguring service iot to use http://localhost:4566
Reconfiguring service iotanalytics to use http://localhost:4566
Reconfiguring service iotevents to use http://localhost:4566
Reconfiguring service iot-data to use http://localhost:4566
Reconfiguring service iot-jobs-data to use http://localhost:4566
Reconfiguring service kafka to use http://localhost:4566
Reconfiguring service kinesis to use http://localhost:4566
Reconfiguring service kinesisanalytics to use http://localhost:4566
Reconfiguring service kms to use http://localhost:4566
Reconfiguring service lambda to use http://localhost:4566
Reconfiguring service logs to use http://localhost:4566
Reconfiguring service mediastore to use http://localhost:4566
Reconfiguring service neptune to use http://localhost:4566
Reconfiguring service organizations to use http://localhost:4566
Reconfiguring service qldb to use http://localhost:4566
Reconfiguring service rds to use http://localhost:4566
Reconfiguring service redshift to use http://localhost:4566
Reconfiguring service route53 to use http://localhost:4566
Reconfiguring service s3 to use http://localhost:4566
Reconfiguring service s3control to use http://localhost:4566
Reconfiguring service sagemaker to use http://localhost:4566
Reconfiguring service sagemaker-runtime to use http://localhost:4566
Reconfiguring service secretsmanager to use http://localhost:4566
Reconfiguring service ses to use http://localhost:4566
Reconfiguring service sns to use http://localhost:4566
Reconfiguring service sqs to use http://localhost:4566
Reconfiguring service ssm to use http://localhost:4566
Reconfiguring service stepfunctions to use http://localhost:4566
Reconfiguring service sts to use http://localhost:4566
Reconfiguring service timestream to use http://localhost:4566
Reconfiguring service transfer to use http://localhost:4566
Reconfiguring service xray to use http://localhost:4566
service: FoobarService
frameworkVersion: '3'
provider:
  name: aws
  runtime: python3.8
  stage: local
  region: us-east-1
  versionFunctions: true
plugins:
  - serverless-localstack
custom:
  defaultStage: local
  profile: default
  localstack:
    debug: true
    stages:
      - local
    host: http://localhost
    edgePort: 4566
    autostart: true
    lambda:
      mountCode: false
    docker:
      sudo: false
functions:
  foo:
    name: foo-local
    handler: handler.foobar
    events: []
  bar:
    name: bar-local
    handler: handler.foobar
    events: []
  foobar:
    name: foobar-local
    handler: handler.foobar
    events:
      - cloudwatchLog:
          logGroup: /aws/lambda/foo-local
          filter: '{ $.level = "error" }'
      - cloudwatchLog:
          logGroup: /aws/lambda/bar-local
          filter: '{ $.level = "error" }'
$ sls deploy --stage local
Running "serverless" from node_modules
config.options_stage: local
serverless.service.custom.stage: undefined
serverless.service.provider.stage: local
config.stage: local
Using serverless-localstack
Reconfiguring service acm to use http://localhost:4566
Reconfiguring service amplify to use http://localhost:4566
Reconfiguring service apigateway to use http://localhost:4566
Reconfiguring service apigatewayv2 to use http://localhost:4566
Reconfiguring service application-autoscaling to use http://localhost:4566
Reconfiguring service appsync to use http://localhost:4566
Reconfiguring service athena to use http://localhost:4566
Reconfiguring service autoscaling to use http://localhost:4566
Reconfiguring service batch to use http://localhost:4566
Reconfiguring service cloudformation to use http://localhost:4566
Reconfiguring service cloudfront to use http://localhost:4566
Reconfiguring service cloudsearch to use http://localhost:4566
Reconfiguring service cloudtrail to use http://localhost:4566
Reconfiguring service cloudwatch to use http://localhost:4566
Reconfiguring service cloudwatchlogs to use http://localhost:4566
Reconfiguring service codecommit to use http://localhost:4566
Reconfiguring service cognito-idp to use http://localhost:4566
Reconfiguring service cognito-identity to use http://localhost:4566
Reconfiguring service docdb to use http://localhost:4566
Reconfiguring service dynamodb to use http://localhost:4566
Reconfiguring service dynamodbstreams to use http://localhost:4566
Reconfiguring service ec2 to use http://localhost:4566
Reconfiguring service ecr to use http://localhost:4566
Reconfiguring service ecs to use http://localhost:4566
Reconfiguring service eks to use http://localhost:4566
Reconfiguring service elasticache to use http://localhost:4566
Reconfiguring service elasticbeanstalk to use http://localhost:4566
Reconfiguring service elb to use http://localhost:4566
Reconfiguring service elbv2 to use http://localhost:4566
Reconfiguring service emr to use http://localhost:4566
Reconfiguring service es to use http://localhost:4566
Reconfiguring service events to use http://localhost:4566
Reconfiguring service firehose to use http://localhost:4566
Reconfiguring service glacier to use http://localhost:4566
Reconfiguring service glue to use http://localhost:4566
Reconfiguring service iam to use http://localhost:4566
Reconfiguring service iot to use http://localhost:4566
Reconfiguring service iotanalytics to use http://localhost:4566
Reconfiguring service iotevents to use http://localhost:4566
Reconfiguring service iot-data to use http://localhost:4566
Reconfiguring service iot-jobs-data to use http://localhost:4566
Reconfiguring service kafka to use http://localhost:4566
Reconfiguring service kinesis to use http://localhost:4566
Reconfiguring service kinesisanalytics to use http://localhost:4566
Reconfiguring service kms to use http://localhost:4566
Reconfiguring service lambda to use http://localhost:4566
Reconfiguring service logs to use http://localhost:4566
Reconfiguring service mediastore to use http://localhost:4566
Reconfiguring service neptune to use http://localhost:4566
Reconfiguring service organizations to use http://localhost:4566
Reconfiguring service qldb to use http://localhost:4566
Reconfiguring service rds to use http://localhost:4566
Reconfiguring service redshift to use http://localhost:4566
Reconfiguring service route53 to use http://localhost:4566
Reconfiguring service s3 to use http://localhost:4566
Reconfiguring service s3control to use http://localhost:4566
Reconfiguring service sagemaker to use http://localhost:4566
Reconfiguring service sagemaker-runtime to use http://localhost:4566
Reconfiguring service secretsmanager to use http://localhost:4566
Reconfiguring service ses to use http://localhost:4566
Reconfiguring service sns to use http://localhost:4566
Reconfiguring service sqs to use http://localhost:4566
Reconfiguring service ssm to use http://localhost:4566
Reconfiguring service stepfunctions to use http://localhost:4566
Reconfiguring service sts to use http://localhost:4566
Reconfiguring service timestream to use http://localhost:4566
Reconfiguring service transfer to use http://localhost:4566
Reconfiguring service xray to use http://localhost:4566

Deploying FoobarService to stage local (us-east-1)
config.options_stage: local
serverless.service.custom.stage: undefined
serverless.service.provider.stage: local
config.stage: local
config.options_stage: local
serverless.service.custom.stage: undefined
serverless.service.provider.stage: local
config.stage: local
Using custom endpoint for STS: http://localhost:4566
Using custom endpoint for CloudFormation: http://localhost:4566
Using custom endpoint for CloudFormation: http://localhost:4566
Using custom endpoint for CloudFormation: http://localhost:4566
Using custom endpoint for CloudFormation: http://localhost:4566
Using custom endpoint for CloudFormation: http://localhost:4566
Using custom endpoint for CloudFormation: http://localhost:4566
Using custom endpoint for S3: http://localhost:4566
Using custom endpoint for S3: http://localhost:4566
Using custom endpoint for Lambda: http://localhost:4566
Using custom endpoint for Lambda: http://localhost:4566
Using custom endpoint for Lambda: http://localhost:4566
Using custom endpoint for CloudWatchLogs: http://localhost:4566
Using custom endpoint for CloudWatchLogs: http://localhost:4566
Using custom endpoint for S3: http://localhost:4566
Using custom endpoint for S3: http://localhost:4566
Using custom endpoint for S3: http://localhost:4566
Skipping template validation: Unsupported in Localstack
Using custom endpoint for CloudFormation: http://localhost:4566
Using custom endpoint for CloudFormation: http://localhost:4566
Overriding S3 templateUrl to http://localhost:4566
Using custom endpoint for CloudFormation: http://localhost:4566
Using custom endpoint for CloudFormation: http://localhost:4566
Using custom endpoint for CloudFormation: http://localhost:4566
Using custom endpoint for CloudFormation: http://localhost:4566
Using custom endpoint for CloudFormation: http://localhost:4566
Using custom endpoint for S3: http://localhost:4566

✔ Service deployed to stack FoobarService-local (14s)

functions:
  foo: foo-local (926 B)
  bar: bar-local (926 B)
  foobar: foobar-local (926 B)
$ sls invoke -f foobar --stage local
Running "serverless" from node_modules
config.options_stage: local
serverless.service.custom.stage: undefined
serverless.service.provider.stage: local
config.stage: local
Using serverless-localstack
Reconfiguring service acm to use http://localhost:4566
Reconfiguring service amplify to use http://localhost:4566
Reconfiguring service apigateway to use http://localhost:4566
Reconfiguring service apigatewayv2 to use http://localhost:4566
Reconfiguring service application-autoscaling to use http://localhost:4566
Reconfiguring service appsync to use http://localhost:4566
Reconfiguring service athena to use http://localhost:4566
Reconfiguring service autoscaling to use http://localhost:4566
Reconfiguring service batch to use http://localhost:4566
Reconfiguring service cloudformation to use http://localhost:4566
Reconfiguring service cloudfront to use http://localhost:4566
Reconfiguring service cloudsearch to use http://localhost:4566
Reconfiguring service cloudtrail to use http://localhost:4566
Reconfiguring service cloudwatch to use http://localhost:4566
Reconfiguring service cloudwatchlogs to use http://localhost:4566
Reconfiguring service codecommit to use http://localhost:4566
Reconfiguring service cognito-idp to use http://localhost:4566
Reconfiguring service cognito-identity to use http://localhost:4566
Reconfiguring service docdb to use http://localhost:4566
Reconfiguring service dynamodb to use http://localhost:4566
Reconfiguring service dynamodbstreams to use http://localhost:4566
Reconfiguring service ec2 to use http://localhost:4566
Reconfiguring service ecr to use http://localhost:4566
Reconfiguring service ecs to use http://localhost:4566
Reconfiguring service eks to use http://localhost:4566
Reconfiguring service elasticache to use http://localhost:4566
Reconfiguring service elasticbeanstalk to use http://localhost:4566
Reconfiguring service elb to use http://localhost:4566
Reconfiguring service elbv2 to use http://localhost:4566
Reconfiguring service emr to use http://localhost:4566
Reconfiguring service es to use http://localhost:4566
Reconfiguring service events to use http://localhost:4566
Reconfiguring service firehose to use http://localhost:4566
Reconfiguring service glacier to use http://localhost:4566
Reconfiguring service glue to use http://localhost:4566
Reconfiguring service iam to use http://localhost:4566
Reconfiguring service iot to use http://localhost:4566
Reconfiguring service iotanalytics to use http://localhost:4566
Reconfiguring service iotevents to use http://localhost:4566
Reconfiguring service iot-data to use http://localhost:4566
Reconfiguring service iot-jobs-data to use http://localhost:4566
Reconfiguring service kafka to use http://localhost:4566
Reconfiguring service kinesis to use http://localhost:4566
Reconfiguring service kinesisanalytics to use http://localhost:4566
Reconfiguring service kms to use http://localhost:4566
Reconfiguring service lambda to use http://localhost:4566
Reconfiguring service logs to use http://localhost:4566
Reconfiguring service mediastore to use http://localhost:4566
Reconfiguring service neptune to use http://localhost:4566
Reconfiguring service organizations to use http://localhost:4566
Reconfiguring service qldb to use http://localhost:4566
Reconfiguring service rds to use http://localhost:4566
Reconfiguring service redshift to use http://localhost:4566
Reconfiguring service route53 to use http://localhost:4566
Reconfiguring service s3 to use http://localhost:4566
Reconfiguring service s3control to use http://localhost:4566
Reconfiguring service sagemaker to use http://localhost:4566
Reconfiguring service sagemaker-runtime to use http://localhost:4566
Reconfiguring service secretsmanager to use http://localhost:4566
Reconfiguring service ses to use http://localhost:4566
Reconfiguring service sns to use http://localhost:4566
Reconfiguring service sqs to use http://localhost:4566
Reconfiguring service ssm to use http://localhost:4566
Reconfiguring service stepfunctions to use http://localhost:4566
Reconfiguring service sts to use http://localhost:4566
Reconfiguring service timestream to use http://localhost:4566
Reconfiguring service transfer to use http://localhost:4566
Reconfiguring service xray to use http://localhost:4566
config.options_stage: local
serverless.service.custom.stage: undefined
serverless.service.provider.stage: local
config.stage: local
config.options_stage: local
serverless.service.custom.stage: undefined
serverless.service.provider.stage: local
config.stage: local
Using custom endpoint for Lambda: http://localhost:4566
{
    "foo": "bar"
}

Limitations

Be aware that following this solution means that defining other events using the YAML syntax won't work. The reason is that YAML does not fully support merging arrays at the moment (related threads: 1, 2).

For instance, the following code

...

cloudwatch_log_events: &cloudwatch_log_events
  ${file(_serverless/cloudwatch_log_events.js)}

...

     events:
       - *cloudwatch_log_events
       - cloudwatchLog:
           ...

would produce the config as follows

     events:
       - - cloudwatchLog:
             ...
       - cloudwatchLog:

Notice the nested list (- -). In order for the config to be valid, there would need to be an option to extend the cloudwatch_log_events array, which is not possible today.

However, to handle different categories of CloudWatch log events (e.g. per service), you can go ahead with granularity on the JavaScript level. Following the above example:

(New) Project structure

.
├── _serverless
│   ├── cloudwatch_log_events.js
│   └── services
│       ├── bar.js
│       └── foo.js
├── handler.py
└── serverless.yml

_serverless/services/foo.js

const getConfig = async (resolveVariable) => {
  const stage = await resolveVariable('sls:stage');
  const logGroups = [
    `/aws/lambda/foo-${stage}-1`,
    `/aws/lambda/foo-${stage}-2`,
  ];
  const logGroupFilter = '{ $.level = "error" }';

  return logGroups.map(function (logGroup) {
    return {
      cloudwatchLog: {
          logGroup: logGroup,
          filter: logGroupFilter
      }
    }
  });
}


module.exports = {
    getConfig: getConfig
}

_serverless/services/bar.js

const getConfig = async (resolveVariable) => {
  const stage = await resolveVariable('sls:stage');
  const logGroups = [
    `/aws/lambda/bar-${stage}-1`,
    `/aws/lambda/bar-${stage}-2`,
  ];
  const logGroupFilter = '{ $.level = "error" }';

  return logGroups.map(function (logGroup) {
    return {
      cloudwatchLog: {
          logGroup: logGroup,
          filter: logGroupFilter
      }
    }
  });
}


module.exports = {
    getConfig: getConfig
}

_serverless/cloudwatch_log_events.js

const foo = require('./services/foo.js');
const bar = require('./services/bar.js');


module.exports = async ({ options, resolveVariable }) => {
    const fooConfig = await foo.getConfig(resolveVariable);
    const barConfig = await bar.getConfig(resolveVariable);

    return [
        ...fooConfig,
        ...barConfig,
    ];
}

The rest of the files remain untouched.

pbajsarowicz
  • 542
  • 6
  • 12