8

Error: Secrets Manager cannot invoke the specified Lambda function. Ensure that the function policy grants access to the principal secretsmanager.amazonaws.com

I'm using Secret Manager to store my key for verifying JWTs.
My planned configuration is to rotate deprecate the keys with the following logic:

my secret looks like this:

{
  current:'my-current-secret',
  previous:'my-previous-secret',
  alg:'encoding alg',
}

*It seemed like overkill to use two secrets and rotating them -- I'm only keeping a memory of the previous token to handle fringe cases for a hand-off. If auth fails I'll check if it verifies with the previous, if it does it'll return an updated cookie using the current key

createSecret:

putSecretValue({
      current: getRandomPassword(...),
      previous: getSecretValue(...)['current'],
      alg: env.param ? env.param : getSecretValue(...)['alg']
})

I'm not using setSecret, testSecret, finishSecret

I'm not using serverless (I will at some point, but I wanted to familiarize myself w/ AWS/GUI first before short-cutting w/ the CLI) I've looked at:

I can't figure out what IAM setting I'm missing.

When I try to set the rotation lambda: add rotation

This flashes (So quickly, I had to record my screen to take a look): false hope

And I immediately get the following error: error msg

I started by giving the lambda full control of secrets manager and lambdas to work backwards into minimal controls, but even throwing the kitchen sink at it I couldn't get it to work:

{
  "permissionsBoundary": {},
  "roleName": "secrets_manager-role-REDACTED",
  "policies": [
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
              "secretsmanager:GetRandomPassword",
              "secretsmanager:CreateSecret",
              "secretsmanager:ListSecrets"
            ],
            "Resource": "*"
          },
          {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "secretsmanager:*",
            "Resource": "arn:aws:secretsmanager:us-east-1:REDACTED:secret:REDACTED"
          }
        ]
      },
      "name": "ReadWriteREDACTEDSecret",
      "id": "REDACTED",
      "type": "managed",
      "arn": "arn:aws:iam::REDACTED:policy/ReadWriteREDACTEDSecret"
    },
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
              "lambda:InvokeFunction",
              "lambda:InvokeAsync"
            ],
            "Resource": "arn:aws:lambda:us-east-1:REDACTED:function:secrets_manager"
          }
        ]
      },
      "name": "invoke_secrets_manager_lambda",
      "id": "REDACTED",
      "type": "managed",
      "arn": "arn:aws:iam::REDACTED:policy/invoke_secrets_manager_lambda"
    },
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:us-east-1:REDACTED:*"
          },
          {
            "Effect": "Allow",
            "Action": [
              "logs:CreateLogStream",
              "logs:PutLogEvents"
            ],
            "Resource": [
              "arn:aws:logs:us-east-1:REDACTED:log-group:/aws/lambda/secrets_manager:*"
            ]
          }
        ]
      },
      "name": "AWSLambdaBasicExecutionRole-REDACTED",
      "id": "REDACTED",
      "type": "managed",
      "arn": "arn:aws:iam::REDACTED:policy/service-role/AWSLambdaBasicExecutionRole-REDACTED"
    },
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "cloudformation:DescribeChangeSet",
              "cloudformation:DescribeStackResources",
              "cloudformation:DescribeStacks",
              "cloudformation:GetTemplate",
              "cloudformation:ListStackResources",
              "cloudwatch:*",
              "cognito-identity:ListIdentityPools",
              "cognito-sync:GetCognitoEvents",
              "cognito-sync:SetCognitoEvents",
              "dynamodb:*",
              "ec2:DescribeSecurityGroups",
              "ec2:DescribeSubnets",
              "ec2:DescribeVpcs",
              "events:*",
              "iam:GetPolicy",
              "iam:GetPolicyVersion",
              "iam:GetRole",
              "iam:GetRolePolicy",
              "iam:ListAttachedRolePolicies",
              "iam:ListRolePolicies",
              "iam:ListRoles",
              "iam:PassRole",
              "iot:AttachPrincipalPolicy",
              "iot:AttachThingPrincipal",
              "iot:CreateKeysAndCertificate",
              "iot:CreatePolicy",
              "iot:CreateThing",
              "iot:CreateTopicRule",
              "iot:DescribeEndpoint",
              "iot:GetTopicRule",
              "iot:ListPolicies",
              "iot:ListThings",
              "iot:ListTopicRules",
              "iot:ReplaceTopicRule",
              "kinesis:DescribeStream",
              "kinesis:ListStreams",
              "kinesis:PutRecord",
              "kms:ListAliases",
              "lambda:*",
              "logs:*",
              "s3:*",
              "sns:ListSubscriptions",
              "sns:ListSubscriptionsByTopic",
              "sns:ListTopics",
              "sns:Publish",
              "sns:Subscribe",
              "sns:Unsubscribe",
              "sqs:ListQueues",
              "sqs:SendMessage",
              "tag:GetResources",
              "xray:PutTelemetryRecords",
              "xray:PutTraceSegments"
            ],
            "Resource": "*"
          }
        ]
      },
      "name": "AWSLambdaFullAccess",
      "id": "REDACTED",
      "type": "managed",
      "arn": "arn:aws:iam::aws:policy/AWSLambdaFullAccess"
    }
  ],
  "trustedEntities": [
    "secretsmanager.amazonaws.com",
    "lambda.amazonaws.com"
  ]
}

My lambda trust policy is as follows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "secretsmanager.amazonaws.com",
          "lambda.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
Schalton
  • 2,867
  • 2
  • 32
  • 44

2 Answers2

15

After combing through the aws gui for ~10 hours and googling extensively I came across a s/o post for a different resource that linked to the docs saying that a lambda's function policy cannot be set in the gui.

I ran the following command in the cli and everything worked:

aws lambda add-permission \
          --function-name secrets_manager \
          --principal secretsmanager.amazonaws.com \
          --action lambda:InvokeFunction \
          --statement-id SecretsManagerAccess

--function-name secrets_manager is because my lambda function is named secrets_manager

source: https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-create-generic-template.html

Schalton
  • 2,867
  • 2
  • 32
  • 44
  • 1
    Sorry to hear that it took you 10 hours :( . Yea Lambda should support updating this function policy in the console. – committedandroider Nov 18 '19 at 08:25
  • Hi @Schalton I wish I could see your answer earlier... I've spent several days on this issue, just wondering if I can use the same command to grant permission if I use the rotation function template for MySQL engine? (link: https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/blob/master/SecretsManagerRDSMySQLRotationSingleUser/lambda_function.py) I got the same permission error if I configure in Secret Manager console directly, so I'm trying to create rotation function manually by using this template, any suggestions? – wawawa Jan 30 '20 at 10:03
  • 1
    @Cecilia sorry, I don't know about that use case specifically. After I ran into this problem, I started doing way more config setting in the CLI (versus the console); I'd recommend looking for CLI solutions since we've learned it is more feature rich -- good luck! If you find a solution please post a Q&A on S/O to help future users (probably me in 6 months haha) – Schalton Feb 01 '20 at 00:27
1

Update for 2023. It looks like a resource-level policy for Lambda functions can now be set in the AWS Console GUI.

Go to your Lambda function, then Configuration > Permissions. Scroll down to the "Resource-based policy statements". Here you can add a permission for an AWS Service (choose Secrets Manager), enter your statement ID, and choose lambda:InvokeFunction for the action. The correct Principal is filled in when you choose the Secrets Manager service.

Lambda - Add permissions - Policy statement

At least your error in Secrets Manager was more descriptive. All I got was "Failed to rotate the secret for my_function", no additional details.

JavaJudt
  • 787
  • 7
  • 8