22

I've created a secret and updated it to have a lambda rotation function

My secret looks like

aws secretsmanager list-secret-version-ids --secret-id envir/username
{
    "Versions": [
        {
            "VersionId": "90179cd3-daa1-48e4-9fe5-dde0a4cf22e4",
            "VersionStages": [
                "AWSPREVIOUS"
            ],
            "LastAccessedDate": 1524528000.0,
            "CreatedDate": 1524568488.358
        },
        {
            "VersionId": "60576823-5d98-4360-af53-7e1f909b88d0",
            "VersionStages": [
                "AWSCURRENT"
            ],
            "LastAccessedDate": 1524528000.0,
            "CreatedDate": 1524568827.466
        }
    ],
    "ARN": "arn:aws:secretsmanager:eu-west-1:8282828282828:secret:username-YdgbPA",
    "Name": "envir/username"
}

and when i try to rotate it, i get this error

An error occurred (InvalidRequestException) when calling the RotateSecret operation: A previous rotation isn’t complete. That rotation will be reattempted.

I can rotate the secret without issues if i trigger the lambda function without issues.

Anyone has any ideas ?

related links:

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
user2599522
  • 3,005
  • 2
  • 23
  • 40

5 Answers5

24

Just a note for people in future who might get the same error...

If you are using the AWS Secrets Manager to rotate an Amazon RDS password, the Secrets Manager will automatically create a Lambda function. This function requires:

  • Access to the Internet (to call the Secrets Manager) OR VPC endpoint for Secrets Manager service in subnet/subnets associated with the lambda function
  • Access to the RDS instance (to login and change the password)

As such, the following combinations work:

  • Publicly accessible database (bad for security) with a Lambda function that is not attached to a VPC, OR
  • The Lambda function in a private subnet with a NAT Gateway in the public subnet (so the Lambda function can access the Internet) OR an Elastic IP Address attached to the Lambda function's ENI

Also, the Security Group attached to the database needs to permit inbound access from the Lambda function. By default, the Lambda function is assigned the same security group as used by the database, so either:

  • Edit the database security group to permit inbound connections from itself (that is, from Lambda to the database via the same security group), OR
  • Change the security group that is used by the Lambda function to one that is currently permitted to access the database security group
Community
  • 1
  • 1
John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • Shouldn't it be possible to just put the rotation lambda and database in the same security group? – Dan Jun 08 '20 at 22:07
  • 1
    @Dan Resources typically do not go "in the same security group". A security group is not like a subnet where resources are "in the subnet". Rather, security group rules apply to each individual resources to which they are associated. Just because two resources have the same security group doesn't mean that they can communicate — this is only possible if the security group has a rule that specifically allows 'itself' to access 'itself'. Thus, it is normally best to put different security groups on different types of resources, and then refer to each other (eg DB allows inbound from Lambda). – John Rotenstein Jun 08 '20 at 22:26
  • +1, appreciate the clarification. Have you gotten this to work in CDK? Currently trying to stand up a serverless aurora stack with auto-key rotation using CDK and banging my head against a wall – Dan Jun 09 '20 at 03:47
  • Feel free to create a new Question with all the details. – John Rotenstein Jun 09 '20 at 04:30
  • I just answered a similar question that also presented itself with 'A previous rotation isn't complete'. John's answer here helped me down the path to the solution I arrived at. https://stackoverflow.com/a/62975607/2344607. Just commenting here in case it helps someone else. – theJasonHall Jul 19 '20 at 02:25
24

For anyone still having this issue what you can try doing is clearing the pending version and reattempting the rotation

For example with a secret with secret id thefrog, call 

aws secretsmanager get-secret-value \
    --secret-id thefrog \
    --version-stage AWSPENDING

to get the version id of the version with the pending label. The result would look like

{                                                                      
    "CreatedDate": 1541540242.561,                         
    "Name": "thefrog",                
    "VersionStages": [                               
        "AWSPENDING"                                        
    ],                                                    
    "SecretString": "TOP-SECRET",                                                    
    "ARN": "arn:aws:secretsmanager:xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "VersionId": "2a27cecb-23c7-4320-b168-78661c24612f"   
}

Then call 

aws secretsmanager update-secret-version-stage \
    --secret-id thefrog \
    --version-stage AWSPENDING \
    --remove-from-version-id 2a27cecb-23c7-4320-b168-78661c24612f

to remove the version of secret that has the pending label.

From here you can retry the rotation

peterm
  • 91,357
  • 15
  • 148
  • 157
committedandroider
  • 8,711
  • 14
  • 71
  • 126
  • Just wondering what ouput indicates the rotation occurs successfully? I got this ```"Versions": [ { "VersionId": "xxx", "VersionStages": [ "AWSCURRENT" ], "LastAccessedDate": 1580342400.0, "CreatedDate": 1580391706.854 }, { "VersionId": "xxxx", "VersionStages": [ "AWSPREVIOUS" ], "LastAccessedDate": 1580342400.0, "CreatedDate": 1580297852.964 }``` – wawawa Jan 30 '20 at 15:20
  • (Sorry can't fit in one comment)Does this mean my credentials have been rotated? I checked the secret in SM console the password hasn't changed. – wawawa Jan 30 '20 at 15:21
  • If you're using a lambda to do the rotation, check its logs. You may find an error that is preventing the rotation from succeeding. – sandinmyjoints Oct 26 '22 at 13:47
  • Excellent. Would you go so far as to incorporate this into the rotation lambda, and if so how? – jtlz2 Oct 30 '22 at 21:48
14

For anyone who thinks that the link at https://forums.aws.amazon.com/thread.jspa?threadID=280093&tstart=0 doesn't apply, make sure to check the output of both aws secretsmanager list-secret-version-ids and aws secretsmanager list-secrets to make sure they are in sync with each other. I just had one secret I could not rotate, kept getting the "A previous rotation isn’t complete. That rotation will be reattempted" error message. I had a support case with AWS open on it, and while I was waiting on hold to speak to a support rep, I decided to check the output of list-secrets, and lo and behold I found an AWSPENDING label on the secret I could not rotate (that label did NOT show up on the output of list-secret-version-ids for that secret). Once I cleared that label, I could then successfully rotate the secret I was having problems with.

subzero2000
  • 151
  • 4
2

The problem was that the lambda function was failing and the retry was happening w/o my control (there is currently no way to limit the retries on a lambda function).

user2599522
  • 3,005
  • 2
  • 23
  • 40
0

I had a similar problem. For my documentdb I have a cloudformation template which got the following template content:

  MyDocumentDbSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: "/secrets/documentdb/root"
      Description: 'DocDB root secret'
      GenerateSecretString:
        SecretStringTemplate: !Sub '{"username": "${DefaultDocDbUser}"}'
        GenerateStringKey: "password"
        PasswordLength: 16
        ExcludeCharacters: '"@/\'

But with this cloudformation template I always get a connection timeout in the lambda function (Which try to change my password for the user).

But when I change my cloudformation template to this with ssl = true in the SecretStringTemplate attribute:

  DocDBClusterRotationSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: "/secrets/documentdb/root"
      Description: 'DocDB root secret'
      GenerateSecretString:
        SecretStringTemplate: !Sub '{"username": "${DefaultDocDbUser}", "ssl": "true"}'
        GenerateStringKey: "password"
        PasswordLength: 16
        ExcludeCharacters: '"@/\'

then it works properly. For my cloudformation type: AWS::SecretsManager::SecretTargetAttachment does not provide my secret with the attributes ssl=true, therefore I need to add it manually in my cloudformation template. Now it works perfectly with no errors.

My secret string looks like this nowadays:

{
 "password": "My PW",
 "engine": "mongo",
 "port": 27017,
 "host": "My Host",
 "ssl": "true",
 "username": "My User"
}
Guchelkaben
  • 1,205
  • 1
  • 12
  • 18