Attempting to decrypt ciphertext within a Lambda function using KMS results in timeout

Question

When decrypting ciphertext from the command line using the AWS CLI, the ciphertext gets decrypted without issues:

$ aws kms decrypt --ciphertext-blob fileb://encrypted-secrets --output text --query Plaintext --region us-east-1 | base64 --decode > decryped-secrets

This decryption operation also works locally when attempting to do so from a js script:

#!/usr/local/bin/node

const fs = require('fs');
const AWS = require('aws-sdk');
const kms = new AWS.KMS({region:'us-east-1'});

const secretPath = './encrypted-secrets';
const encryptedSecret = fs.readFileSync(secretPath);

const params = {
      CiphertextBlob: encryptedSecret
};

kms.decrypt(params, function(err, data) {
  if (err) {
    console.log(err, err.stack);
  } else {
    const decryptedScret = data['Plaintext'].toString();
    console.log('decrypted secret', decryptedScret);
  }
});

However, when attempting to do so with almost the same exact code as above from within the context of an AWS Lambda function, the invocation of the function results in a timeout:

'use strict';

const zlib = require('zlib');
const mysql = require('mysql');
const fs = require('fs');
const AWS = require('aws-sdk');
const kms = new AWS.KMS({region:'us-east-1'});

const secretPath = './encrypted-secrets';
const encryptedSecret = fs.readFileSync(secretPath);

const params = {
    CiphertextBlob: encryptedSecret
};

exports.handler = (event, context, callback) => {
    kms.decrypt(params, (err, data) => {
       if (err) {
            console.log(err, err.stack);
            return callback(err);
        } else {
            const decryptedScret = data['Plaintext'].toString();
            console.log('decrypted secret', decryptedScret);
            return callback(null, `Successfully processed ${parsed.logEvents.length} log events.`);
        }
    });
};

timeout log:

START RequestId: start-request-id-redacted Version: $LATEST
END RequestId: end-request-id-redacted
REPORT RequestId: report-requested-id-redacted  Duration: 10002.43 ms   Billed Duration: 10000 ms   Memory Size: 128 MB Max Memory Used: 18 MB  
2016-11-13T19:22:28.774Z task-id-redacted Task timed out after 10.00 seconds

Notes:

• If I comment out the call to kms.decrypt and attempt to console.log the params or anything really, the values are output without issues. There seems to be some sort of issue with the kms.decrypt call, and no actual error beyond the timeout is returned.
• The policy attached to the role under which the lambda function is invoked contains the attached policy AWSLambdaVPCAccessExecutionRole, and also the following attached inline policy:

policygen-lambda_basic_execution_and_kms_decrypt-201611131221:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "sid-redacted",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": [
                "arn:aws:kms:us-east-1:account-redacted:key/key-id-redacted"
            ]
        }
    ]
}

• I've redacted any identifying information from the code.

Answer 1

After some thorough conversations with AWS support folks, whom have been very helpful, we have an answer:

The primary reason why there was a timeout was due to a lack of connectivity from within the Lambda function to the KMS service, due to the KMS service not having an endpoint in the VPC where the Lambda function was configured.

In order for a Lambda function in a VPC to connect to any service other than Amazon S3, which does have an endpoint in the VPC, the Lambda function has to be situated in/associated with at least one, but preferably two private subnets, with their routing tables including a destination route of 0.0.0.0/16 to a NAT Gateway.

It is not possible to have the Lambda function be in a public subnet, with an Internet Gateway.

Steps to getting a VPC-bound Lambda function to access KMS and all other services that don't have VPC endpoints:

1. Create or take note of an existing Private Subnet, which has a routing table entry for 0.0.0.0/0 to a NAT Gateway.
   • If you don't already have a NAT Gateway, a Routing Table, and the Subnet, as specified above, you'll have to create and associate them with each other appropriately first.
2. Attach the Lambda function to the private subnets above, when creating the Lambda function, or edit the Lambda function to have that configuration.

If you follow those two steps, you should be able to invoke kms.encrypt and other requests from within your Lambda function, which require outbound/egress internet connectivity, due to those services not having endpoints within your VPC.

Answer 2

EC2 instances come with their own public IP by default so they have no issues accessing any services requiring access to the internet (such as KMS).

Lambda functions attached to your VPC don't have a public IP so to access a service via the internet (such as KMS) you need a NAT set up just as described by zealoushacker.

Answer 3

To add to zealoushacker's excellent answer, you should also check that your lambda' security group settings have an outbound rule that points to 0.0.0.0 and any port.

In our case, we were already running in private subnets, but had restricted security groups to our RDS database.

Answer 4

Just to do a quick update, I had the same problem but now we can directly add an Endpoint pointing to KMS to the VPC your lambda is attached to.

You can find details here : https://docs.aws.amazon.com/kms/latest/developerguide/kms-vpc-endpoint.html

I do this update since I was about to follow the other answers (which should still work) but this way seems more natural.

Answer 5

Like @killthrush mentioned, aws endpoint is a very good way to implement this.

You just add the endpoint and thats it, nothing more is needed and it just works.

My terraform implementation looks like this:

resource "aws_vpc_endpoint" "kms_endpoint" {
  service_name        = "com.amazonaws.${data.aws_region.current.name}.kms"
  vpc_id              = data.aws_vpc.default.id
  subnet_ids          = data.aws_subnets.default.ids
  vpc_endpoint_type   = "Interface"
  private_dns_enabled = true

  tags = {
    Name  = "${var.env}-kms-endpoint"
  }
}