0

How to clear the DNS cache of an instance in AWS Lambda?

Context:
We backup the database to an external S3 provider. It has worked for few years now.
Recently they did a maintenance and added some proxies servers and renewed their SSL certificates (20 april) and since then (5 days later) I'm still getting the error:

Copying zip failed because: Could not connect to disk s3-infomaniak because: GuzzleHttp\Exception\ConnectException: cURL error 6: Could not resolve host: s3.swiss-backup02.infomaniak.com (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3.swiss-backup02.infomaniak.com/default?list-type=2&prefix=database-backup%2F&delimiter=%2F

We suppose that the DNS cache is still pointing to the old destination.

  • The website is managed with Laravel vapor: https://vapor.laravel.com
  • I confirm the S3 is resolvable from our AWS EC2 and from my machine
  • The backups works from the EC2 and from my machine
  • With or without the NAT, it doesn't change anything
  • Redeploying didn't change anything
Clément Baconnier
  • 5,718
  • 5
  • 29
  • 55
  • 1
    "Could not resolve host" if it's a DNS cache, it's a negative cache. https://serverfault.com/questions/426807/how-long-does-negative-dns-caching-typically-last explains how that's controlled. I see a value of 60 (seconds) in the SOA record for the host in question. I don't think it's DNS caching that's causing this. – erik258 Apr 25 '23 at 17:59
  • I don't have any other idea. It only happens from our AWS lambdas (with and without NAT). I works successfully when running from a VPC or locally. – Clément Baconnier Apr 27 '23 at 06:32
  • Are the lambdas not running in the same VPC that works? – erik258 Apr 27 '23 at 16:22
  • 1
    @erik258 Sorry for the delay, I was looking with AWS support. It seems the issue is either an issue with the vapor's Dockerfile nor infomaniak new proxies / servers. By creating a new lambda, I'm able to reach the domain from the PHP native runtime with cURL, but as soon I switch to vapor's Dockerfile, I'm not able to reach the domain with cURL. At this point I think it's something I can't do anything with, but still looking with Infomaniak support. Thanks for your help ! – Clément Baconnier May 04 '23 at 15:40

1 Answers1

1

I figured out the issue, and it was not the cache.

  1. I could replicate the issue by creating a sub domain with 24 A records (the same amount of records that have been added to s3.swiss-backup02.infomaniak.com)
  2. Then I could still replicate the issue by (and only by) using an Alpine based image on lambda.

Apparently, Alpine seems to have some DNS issues on Kubernetes cluster due to the usage of musl-libc instead of g-libc1. Having a "large" DNS answer seems to be the case with lambda.

Since Laravel vapor is using Alpine based image for their Dockerfiles2 (You're not concerned if you use native runtimes) I did a fork3 of their image with php:8.2-fpm-buster and now the URL is reachable.

1 https://stackoverflow.com/a/65593511/8068675
2 https://github.com/laravel/vapor-dockerfiles/blob/master/php82.Dockerfile
3 https://github.com/melba-ch/vapor-dockerfiles-buster

Clément Baconnier
  • 5,718
  • 5
  • 29
  • 55