4

Recently attached a key from AWS KMS to encrypt CloudWatch log groups for AWS Systems Manager Session Manager. Now I can't connect to any session.

What is this error and how to fix?

Your session has been terminated for the following reasons: ----------ERROR------- Encountered error while initiating handshake. Fetching data key failed: Unable to retrieve data key, Error when decrypting data key AccessDeniedException: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access. status code: 400, request id:

The key definitely exists and worked fine when attaching to the log group. Session Manager was also working fine before trying to encrypt them. Are there extra permissions I need to add to a policy somewhere?

ForeverLearningAndCoding
  • 499
  • 2
  • 7
  • 16
  • Does your KMS have key policies allowing session manager access? – Marcin Jul 01 '20 at 01:37
  • Don't think so - will try now, thanks. Not entirely sure all that's required in the key policy but I'll try to add ssm:StartSession and ssm:TerminateSession and see what happens. – ForeverLearningAndCoding Jul 01 '20 at 01:47
  • Not `ssm:StartSession`, but encryption and decryption permissions with your kms cmk for the session manager. – Marcin Jul 01 '20 at 01:48
  • Check [here](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-preferences-enable-encryption.html): After you enable AWS KMS key encryption for your session data, both the users who start sessions and the instances that they connect to must have permission to use the key. You provide permission to use the CMK with Session Manager through IAM policies. – Marcin Jul 01 '20 at 01:51
  • I checked on this and it seems that SSM with encrypted log groups is actually working in some instances but not in others - without adding these policies. I am not sure why it's working in only some of the instances but they were all set up similarly with CloudFormation templates. The instances that DO work are in public subnets but still - not all of the instances in public subnets are working with Session Manager. – ForeverLearningAndCoding Jul 01 '20 at 03:57
  • What about instance roles? Maybe they are some differences there? – Marcin Jul 01 '20 at 03:59
  • Instances in private subnets won't be able to connect to SSM nor KMS without nat gateway or vpc interface endpoints. – Marcin Jul 01 '20 at 04:00
  • Session Manager was working in private subnets before I turned on encryption with LogGroups, so everything was fine previously. VPC has a NAT, too. It was only after I linked the CMK key that now Session Manager no longer works with some instances. – ForeverLearningAndCoding Jul 01 '20 at 04:11
  • It sounds like permissions issue. The error msg is `AccessDeniedException`. I would double check all the permissions. if some instances work, then have to identify what's different in others that they don't work. – Marcin Jul 01 '20 at 04:14
  • 1
    Figured it out. I had to attach AmazonEC2RoleforSSM to the IAM instance profile. I read that it's better to use AmazonSSMManagedInstanceCore but that wasn't enough permissions for SSM so I changed it to AmazonEC2RoleforSSM and it works now. I fear there may be some security concerns with using AmazonEC2RoleforSSM however... – ForeverLearningAndCoding Jul 01 '20 at 04:18
  • Glad to hear. If was instance role after role. If you don't mined, I would like to provide an answer for future reference? – Marcin Jul 01 '20 at 04:19
  • 1
    Sure, go ahead. – ForeverLearningAndCoding Jul 01 '20 at 04:22

4 Answers4

2

Based on the comments.

The issue was caused by incorrect permissions in the instance role used.

Changing policies in the role from AmazonSSMManagedInstanceCore to AmazonEC2RoleforSSM solved the issue.

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • 1
    `AmazonEC2RoleforSSM` is deprecated now: https://aws.amazon.com/blogs/mt/applying-managed-instance-policy-best-practices/ – davegallant Aug 23 '21 at 18:23
1

Permissions need to be modified.

Select a role from the permission settings of the service you are using You need to add a policy.

Please add the following information to the policy.

{
     "Version": "2012-10-17",
     "Statement": [
         {
             "Effect": "Allow",
             "Action": "kms:Decrypt",
             "Resource": "*"
         }
     ]
}

=======================================================================

In my case, the above error occurred while executing the Lambda function.

I solved it by adding a policy like this:

enter image description here

Jesse Lee
  • 174
  • 1
  • 6
  • This grants permission to decrypt _any_ data anywhere in AWS, which might not be the most secure of ideas... – space55 Jul 16 '21 at 16:59
0

Here is the solution provided by AWS, essentially adding permissions to your instance profile to create encrypted logs on Cloudwatch, of course, you also need to add permissions to Decrypt the session as mentioned in some of the answers above. Be aware that for security reasons you should scope permissions as much as possible.

r4cc00n
  • 1,927
  • 1
  • 8
  • 24
0

An other way this error can appear is if setting up KMS on SSM Session manager to encrypt sessions and if:

  • KMS key is pending allowing use of key (ref) to EC2 IAM role
  • KMS key is pending allow use of key to IAM user/role used for Session manager.
Efren
  • 4,003
  • 4
  • 33
  • 75