How to normalize a private key stored on AWS secrets manager

Question

EDIT: AS OF Feb 2020, AWS SEEMS TO have FIXED THIS BUG. THE BASE64ing and other wise is no longer needed.

I have my secret stored as a string but of course when aws stores the secret it removes white space and line breaks. On top of it it wraps the value in json.

When I run aws secretsmanager get-secret-value --secret-id my-private-key > private.pem it returns something like.

{
    "Name": "ai-data-devops-ansible-deploy-key",
    "VersionId": "fedafe24-d3eb-4964-9a8f-7f4ecb375a35",
    "SecretString": "-----BEGIN RSA PRIVATE KEY-----\nasdkmnasefkljzsdkffjsldkgfjlzkmsdflkNOTAREALKEYasddkjnsfdlzxdfvlkmdggo=\n-----END RSA PRIVATE KEY-----\n",
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": 1568147513.11,
    "ARN": "arn:aws:secretsmanager:us-east-1:13726472r4:secret:my-private-key-XQuwafs"
}

So I need to:

Strip get the value out of json
Reformat the string to be more like

-----BEGIN RSA PRIVATE KEY-----
asdkmnasefkljzsdkffjsldkgfjlzkmsdflkNOTAREALKEYasddkjnsfdlzxdfvlkmdggo=
-----END RSA PRIVATE KEY-----

I wonder whether **AWS Systems Manager Parameter Store** would be a better fit for this type of 'secret'? — John Rotenstein, Oct 18 '19 at 03:49
@JohnRotenstein any reason why AWS System Manager Parameter Store would be a better fit for this secret? AWS Secrets Manager seems to be appropriate for this private key — committedandroider, Oct 20 '19 at 08:06
@Josh Well, since you are experiencing difficulties using Secrets Manager, I figured that Parameter Store might treat you better. — John Rotenstein, Oct 21 '19 at 03:23
I am trying : to store the secret manager as plaintext, and retrieve the string directly without load them as jason, and it works for me. — Emma Y, Feb 21 '20 at 19:47
@EmmaYang. I just retried and it worked with out tghe base64. AWS much have fixed the bug. — Josh Beauregard, Feb 24 '20 at 13:49
@JoshBeauregard, had you implemented SSH key rotation? I'm looking for how to generate SSH private key using a custom lambda function, encode it and then store it as a new secret. — nidhi_007, Nov 25 '21 at 07:21
@nidhi_007 I have but \not in a generalized manner, via lambda. I have a jenkins job and a boto3 library that does it for me, and I am rotating the secret for a Bitbucket user. — Josh Beauregard, Nov 29 '21 at 13:08
I am on `aws-cli/2.9.6 Python/3.11.0 Darwin/22.1.0` and you are right `aws secretsmanager create-secret --name test-newlines --secret-string "$(printf "line1\nline2")"` works. However, I still think `base64 -w0` is a smart move. — Chris Beck, Jan 16 '23 at 00:09

Jason Steele · Answer 1 · 2019-11-08T21:15:09.893

44

Another option would be to base64 encode the PEM for storage:

Encode the key:

$ cat private_key 
-----BEGIN RSA PRIVATE KEY-----
asdkmnasefkljzsdkffjsldkgfjlzkmsdflkNOTAREALKEYasddkjnsfdlzxdfvlkmdggo=
-----END RSA PRIVATE KEY-----
$ base64 private_key > encoded_private_key

$ cat encoded_private_key
LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQphc2RrbW5hc2Vma2xqenNka2ZmanNsZGtnZmpsemttc2RmbGtOT1RBUkVBTEtFWWFzZGRram5zZmRsenhkZnZsa21kZ2dvPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

Get the key back:

$ base64 -D encoded_private_key
-----BEGIN RSA PRIVATE KEY-----
asdkmnasefkljzsdkffjsldkgfjlzkmsdflkNOTAREALKEYasddkjnsfdlzxdfvlkmdggo=
-----END RSA PRIVATE KEY-----

Edit: Assuming the secret is base64 encoded, this would work:

Encode and push:

aws secretsmanager create-secret --name my-private-key --secret-string `base64 private.pem`

Pull and decode:

aws secretsmanager get-secret-value --secret-id my-private-key --query 'SecretString' --output text |base64 -D > private.pem

Doing the --query --output text thing might make it simpler to parse even if you don't want to base64 encode it as well.

edited Nov 08 '19 at 21:15

answered Nov 07 '19 at 16:06

Jason Steele

441
3
4

1

does this eliminate the problem of having to get the value out of the json, I can see that this will eliminate the formatting issues that `sed` fixes? – Josh Beauregard Nov 07 '19 at 17:21
2

You can tell the aws cli to use jsonpath expressions to handle that for you. The full command would be $ aws secretsmanager get-secret-value --secret-id (secret id) --query 'SecretString' --output text |base64 -D – Jason Steele Nov 08 '19 at 03:36
2

Incidentally, the command to push the secret would be: ``aws secretsmanager create-secret --name my-private-key --secret-string `base64 private.pem` ``. – Jason Steele Nov 08 '19 at 03:52
2

On my computer (Ubuntu 18.04), I had to change the above commands to: `aws secretsmanager create-secret --name my-private-key --secret-string "$(base64 private.pem)"` and `aws secretsmanager get-secret-value --secret-id my-private-key --query 'SecretString' --output text | base64 -d > private.pem` – Xuewei Zhang Jul 22 '20 at 20:27
1

i figured aws cli has --file:// and --fileb:// parameters to add content of (binary) files – Hendrik Jander Oct 03 '20 at 13:05
Great solution. I recommend using `base64 -w 0` to avoid having `base64` add newlines. I use this technique to store Kubernetes secrets in AWS Secrets Manager with `kubectl get secret foo -o yaml | base64 -w 0` – Chris Beck Jan 15 '23 at 23:53

score 9 · Accepted Answer · answered Jan 06 '20 at 20:34

I came up with a solution that leveraged storing a secret in secrets manager as plain text.

Store the secret in secrets manager as plain text. They console will have JSON brackets but I removed those.
Use the cli to get the secret output as plain text. Now the \n and \s in the text will be converted to the line breaks and spaces they're supposed to be
```
aws secretsmanager get-secret-value --secret-id privatekey --query 
'SecretString' --output text > private.pem
```

The pem file will now be properly formatted

    -----BEGIN RSA PRIVATE KEY-----
    MIIG3DCCBM
    -----END RSA PRIVATE KEY-----

Josh Beauregard · Answer 3 · 2019-10-21T17:05:55.293

3

You need to Pipe (|) the output through a few steps

To return just the value of the key from the json use jq ".SecretString"
To format the public key use cut -b 2- |tr -d '"' |sed -En "s/\\\n/\n/pg"

This will return what you want to to.

Also note that you will want to make private.pem read only. (chmod 400 private.pem)

In summery the full command will look like:

aws secretsmanager get-secret-value --secret-id my-private-key | jq ".SecretString" |cut -b 2- |tr -d '"' |sed -En "s/\\\n/\n/pg" > private.pem

edited Oct 21 '19 at 17:05

answered Oct 17 '19 at 15:37

Josh Beauregard

2,498
2
20
37

1

Could you update the question with the cli call you used to put the secret in secretsmanager? – committedandroider Oct 20 '19 at 08:05
2

I think I used the console when I was creating it sorry – Josh Beauregard Oct 20 '19 at 23:35
2

oh no worries. I don't think your full command has the jq part. But it be nice if the user didn't have to reformat everything back in the first place :) – committedandroider Oct 21 '19 at 00:05
Use `jq -r` to avoid having to remove quotation marks from the output as a separate step. Otherwise, `jq` is actually printing a JSON-formatted string, which isn't technically what you want. You want the literal string value, which is what `-r` makes it print. – Rob Kennedy Jan 31 '23 at 16:02

score 0 · Answer 4 · answered Dec 13 '21 at 10:29

Recently faced similar "issue".

Brief description

We generated usual public | private {file-name}.pem keys which had format as:

---BEGIN RSA ... KEY---
...
---END RSA ... KEY---

We stored those public | private {file-name}.pem keys in AWS SSM (parameter store) service using SecureString value type.
Afterwards we had to fetch those keys using boto3.

Resolution

--> This piece of code will return you base64 ENcoded representation of your key

from boto3 import client
 
parameter_name = '/ssm/parameter/name/to/fetch'
key = client('ssm').get_parameter(Name=parameter_name).get('Parameter', {}).get('Value', '')

--> This piece of code will return you base64 DEcoded representation of your key

from boto3 import client
 
parameter_name = '/ssm/parameter/name/to/fetch'
key = client('ssm').get_parameter(Name=parameter_name, WithDecryption=True).get('Parameter', {}).get('Value', '')

So, make sure to employ additional parameter as WithDecryption=True within client('ssm').get_parameter() which will automatically solve the decoding of the string.

Reference:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Client.get_parameter

score 0 · Answer 5 · answered Mar 10 '22 at 06:28

Inspired from solution of Jason Steele.

I know that the Issue is raised for AWS-CLI, I have faced similar issue while retrieving the information in Java.

Solution:

While configuring the public/ private key in AWS console, decode the entire key content with Base64 ( You can also use Notepad++ )

While retrieving the data, decode and get it. It resolves the issue. PFB the java code.

secret = getSecretValueResult.getSecretString(); // gets the entire secret object
Object obj=JSONValue.parse(secret); 
        
        //creating an object of JSONObject class and casting the object into JSONObject type  
        JSONObject jsonObject = (JSONObject) obj;   

        //getting values form the JSONObject and casting that values into corresponding types  
        String vendorPublicKey =  (String) jsonObject.get("vendorPublicKey");
  String decodedKey = vendorPublicKey != null ? new String(Base64.getDecoder().decode(vendorPublicKey)) : "";

decodedKey will have the public/private key in valid format.

score 0 · Answer 6 · answered May 12 '23 at 21:21

0

Using plain bash

I know it's an old thread but I was scratching my head for while to get the private-key properly formatted, like this:

and the easiest solution I found, using just plain bash is to use echo -e "$( ... )", like this:

echo -e "$(aws secretsmanager get-secret-value \
               --secret-id <my_secret_id> \
               --query 'SecretString' \
               --output text | grep -o '"<my_key>":"[^"]*' | grep -o '[^"]*$')"

Hope that helps!!

answered May 12 '23 at 21:21

MacUsers

2,091
3
35
56

Do we still need to Base64 encode and store the certificate in secret manger?, The command given is not working. I'm not saving the creds manually but via cloudformation parameter, so does the new lines in certificate are interpreted differently? – Ranyk Aug 23 '23 at 08:54
Well, it was definately working at the time of writing nd that's the way I stored it. I'll have another look into what I actually did or if I had to change anything after this post and report back here soon. – MacUsers Aug 23 '23 at 11:56
@Ranyk - also, what exactly not working re. that command? Did you replace the bits within `` and `` with the actual values of yours? – MacUsers Aug 23 '23 at 11:57

How to normalize a private key stored on AWS secrets manager

6 Answers6

Brief description

Resolution

Reference:

Using plain bash