0

The problem is disappeared after re-deploy the whole AWS CDK Stack. I will keep the question open and update to it if issue occurs again. Speical thanks to @Chetan for helping and not leaving me alone.

I have a .NET6 Web API hosted as AWS Lambda. It retrieves data from the AWS Secret manager using AmazonSecretsManagerClient from Amazon.SecretsManager package. It was working fine until two days ago. To test the connection again, I tried to simply add and run the following code in my Program.cs, however, only the first console.writeline message is printed in log. Then the application just hangs there until reaching Lambda's timesout.

Console.WriteLine("Calling secret manager request - started");
var smClient = new AmazonSecretsManagerClient();
var descriptResponse = await smClient.DescribeSecretAsync(new DescribeSecretRequest
{
    SecretId = "secretName"
});
Console.WriteLine($"got ARN - {descriptResponse.ARN}");
Console.WriteLine("Calling secret manager request - end");

What I have done so far:

  1. I tried to put a try-catch around it, but still no exception was thrown.
  2. I checked the Lambda's IAM role which has the permission - see screenshot 1; I also created a new lambda function with the same role and code, which is working fine that approves the IAM role is correct
  3. I tried other methods like ListSecretsAsync() etc, and the same result - nothing is returned
  4. I tried to deploy a working version (main branch and works in UAT) to my dev instance, which has the same prolbem.

It would be very appreciated if anyone can point me in the right direction to debug it further.

Screnshot 1 enter image description here

Wjun
  • 63
  • 5
  • Is the entire lambda function code is following async/await pattern? – Chetan Jun 22 '22 at 03:07
  • @Chetan I didn't think of this because it was working for a long time until a few days ago (updated question to include this info), is there any method I can test if that's the problem? – Wjun Jun 22 '22 at 03:30
  • `It was working fine until two days ago.` how does it not working now? What is the different behavior you noticed in original code? Do see any error in the original code? – Chetan Jun 22 '22 at 03:32
  • @Chetan it used to run through the whole `Program.cs` and get the secrets correctly, but it suddenly stops at getting the secrets and nothing happen until the function get timeout. I even re-deploy a working version (updated in questions as point 4), which is still not working. It makes me think if the whole resource is broken. I have destroyed the whole AWS stacks and doing a new CDK deploy. Will update the question asap. – Wjun Jun 22 '22 at 03:38
  • I can imagine more information is needed. But I am not sure what and how to grant that information. Any suggestions would be very appreciated. – Wjun Jun 22 '22 at 03:43
  • @Chetan re-deploy the whole AWS CDK stack makes the problem goes away... May I ask how should I deal with this StackOverflow question? Should I just leave it here? – Wjun Jun 22 '22 at 03:48
  • Yes... you may update the question with the latest details.. and if later you find the root cause and solution, you can post that as an answer below so that other folks can benefit from it. – Chetan Jun 22 '22 at 03:51

1 Answers1

0

You almost definitely have the same issue as AWS Lambda access Secrets Manager from within VPC and the answer to that question should apply.

In summary, if your Lambda is in a VPC, it can't connect to Secrets Manager by default and the clue will be the timeout.

In your case a redeploy fixed the issue, which suggests to me that a configuration change in your VPC (or the Secrets Manager VPC endpoint) caused the initial outage.

jbowtie
  • 131
  • 3
  • yes, my lambda is in a VPC. However, it worked before. In regards to my redeploy, the only VPC-related setting that got updated is it created a new security group that connects to the same VPC. Great point though! – Wjun Jun 22 '22 at 05:06