0

I have an AWS Lambda Function that:

  • checks the database and creates notifications
  • send those notifications to Firebase

Because of the database interaction, the function needs to be on a VPC. I followed this guide to give the function a way to connect to Firebase. How do I give internet access to a Lambda function that's connected to an Amazon VPC?. It worked.

The Lambda Function also interacts with an AWS S3 Bucket because the function is deployed using Zappa with slim_handler = True which makes Zappa upload a small handler to Lambda and load the actual project from S3 at runtime.

Some of the notifications are sent and some aren't. When I check logs, for the successfully deliver notification I see the following:

Starting new HTTPS connection (1): fcm.googleapis.com:443
...
https://fcm.googleapis.com:443 "POST /fcm/send HTTP/1.1" 200 None

For the other notifications that are not delivered it logs:

Starting new HTTP connection (1): 169.254.169.254:80
...
Task timed out after 180.01 seconds
Instancing..

AWS Lambda Functions have a retry policy and because of this time out, the function runs 3 times and notifications are triplicated on the database (but not sent to Firebase).

Why is that the HTTP connection starts on 169.254.169.254:80 and not on the correct endpoint that is fcm.googleapis.com:443?

I am not sure if this is relevant, but Zappa automatically sets up an event that runs 4min in order to keep the Lambda function warm.

Why sometimes connecting to Firebase works and sometimes not?

Mark B
  • 183,023
  • 24
  • 297
  • 295
Sofia
  • 445
  • 4
  • 17
  • See the Intermittent Connectivity section at [Why can't an AWS lambda function inside a public subnet in a VPC connect to the internet?](https://stackoverflow.com/a/52994841/271415). – jarmod Oct 19 '22 at 16:30

1 Answers1

3

It sounds like some of the subnets the Lambda function is configured to run in do not have a route to your NAT instance or NAT Gateway. If any of the subnets you have configured for your Lambda function are public subnets, then you will see this issue.

Mark B
  • 183,023
  • 24
  • 297
  • 295
  • I did have 2 subnets, one of them being public and the other private with route to the NAT gateway. I removed the association to the public subnet and one test notification went through so I think it worked! I will do some more tests with more notifications. Thank you for your help!! – Sofia Oct 19 '22 at 13:36
  • Based on your comment and my question https://stackoverflow.com/questions/74126111/aws-lambda-randomly-results-with-timeout, my lambda is assigned to 4 subnets. do you mean that lambda runtime sometimes assigned to a subnet which is not allowed to do API call (firewall things), and sometimes assigned to one that can do API call ? It really makes sense. – aykcandem Oct 19 '22 at 13:37
  • @aykcandem yes that is correct – Mark B Oct 19 '22 at 13:43
  • you were %100 right, just tried it, and worked. thank you very much. I spent several days to figure out what is wrong. Checked my code, API endpoint, limitations everything. Almost gave up, then wrote here, and you pointed out the solution. appreciate @Mark B – aykcandem Oct 19 '22 at 13:47