AWS ACM certificate not validating

Question

Before I begin let me say that I read thoroughly all the stack overflow posts and resources in the appendix, and could not find a solution to my problem.

I am trying to create, validate and connect a subdomain through Route53 and AWS Certificate Manager. The subdomain is challenge.sre.mycompany.com.

The terraform plan looks something like this:

# module.project_challenge.module.challenge-certificate.aws_acm_certificate.cert will be created
  + resource "aws_acm_certificate" "cert" {
      + arn                       = (known after apply)
      + domain_name               = "challenge.sre.mycompany.com"
      + domain_validation_options = [
          + {
              + domain_name           = "challenge.sre.mycompany.com"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
        ]
      + id                        = (known after apply)
      + status                    = (known after apply)
      + subject_alternative_names = (known after apply)
      + tags_all                  = (known after apply)
      + validation_emails         = (known after apply)
      + validation_method         = "DNS"
    }

  # module.project_challenge.module.challenge-certificate.aws_acm_certificate_validation.cert will be created
  + resource "aws_acm_certificate_validation" "cert" {
      + certificate_arn         = (known after apply)
      + id                      = (known after apply)
      + validation_record_fqdns = (known after apply)
    }

  # module.project_challenge.module.challenge-certificate.aws_route53_record.cert["challenge.sre.mycompany.com"] will be created
  + resource "aws_route53_record" "cert" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = (known after apply)
      + records         = (known after apply)
      + ttl             = 60
      + type            = (known after apply)
      + zone_id         = (known after apply)
    }

  # module.project_challenge.module.vpc.aws_route53_zone.public will be created
  + resource "aws_route53_zone" "public" {
      + arn           = (known after apply)
      + comment       = "Managed by Terraform"
      + force_destroy = false
      + id            = (known after apply)
      + name          = "sre.mycompany.com"
      + name_servers  = (known after apply)
      + tags_all      = (known after apply)
      + zone_id       = (known after apply)
    }

As you can see, it create a public hosted zone, an acm certificate and even the validation record. The problem here is that the certificate is stuck on 'Pending Validation` for about 48 hours.

Some details:

The domain is registered through our production account, where I am working on our development account for this.
Both accounts are in the same AWS organisation (if this matters)
Terraform created a public hosted zone sre.mycompany.com with the following attributes:

sre.mycompany.com   NS Records: 
ns-001.awsdns-01.com.
ns-002.awsdns-02.net.
ns-003.awsdns-03.co.uk.
ns-004.awsdns-04.org.

sre.mycompany.com   SOA Simple  Record:
ns-001.awsdns-01.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

CNAME   Simple  Record
_g938534f3gfe03832h34.challenge.sre.mycompany.com   _89432htieh4934hw043f.tkfpekghn.acm-validations.aws.

Obviously the real values are obfuscated*

When I dig sre.mycompany.com or dig challenge.sre.mycompany.com I get:

;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 16577
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

When I dig just mycompany.com I get:

;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61857
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 5

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;mycompany.com.     IN  A

;; ANSWER SECTION:
mycompany.com.  300 IN  A   <some-ip-hidden>

;; AUTHORITY SECTION:
mycompany.com.  169554  IN  NS  ns-555.awsdns-55.com.
mycompany.com.  169554  IN  NS  ns-666.awsdns-66.net.
mycompany.com.  169554  IN  NS  ns-777.awsdns-77.org.
mycompany.com.  169554  IN  NS  ns-888.awsdns-88.co.uk.

Notice that the nameservers here are different from the ones I see in the console of my terraform created hosted zone (scroll above ns-001.awsdns-01.com. etc)

I cannot seem to fetch the CNAME record from my terminal.

In AWS everything seems to work fine on the other hand. When I go to:

Route 53> Hosted zones > Test Record I do get the value of the CNAME record:

Response returned by Route 53 Response from Route 53 based on the following options.

Hosted zone: sre.mycompany.com Record name: _g938534f3gfe03832h34.challenge.

Record type: CNAME DNS response code: No Error Protocol: UDP Response returned by Route 53: _89432htieh4934hw043f.tkfpekghn.acm-validations.aws.

At last if I , the response is:

;; Received 888 bytes from <some-ip-hidden>#53(ns-666.awsdns-66.net) in 3 ms
mycompany.com.  169201  IN  NS  ns-666.awsdns-66.net.
mycompany.com.  169201  IN  NS  ns-777.awsdns-77.org.
mycompany.com.  169201  IN  NS  ns-888.awsdns-88.co.uk.
mycompany.com.  169201  IN  NS  ns-555.awsdns-55.com.
;; BAD (HORIZONTAL) REFERRAL
;; Received 888 bytes from <some-ip-hidden>#53(ns-888.awsdns-88.co.uk) in 4 ms

mycompany.com.  169201  IN  NS  ns-777.awsdns-77.org.
mycompany.com.  169201  IN  NS  ns-666.awsdns-66.net.
mycompany.com.  169201  IN  NS  ns-888.awsdns-88.co.uk.
mycompany.com.  169201  IN  NS  ns-555.awsdns-55.com.
;; BAD (HORIZONTAL) REFERRAL

;; Received 888 bytes from <some-ip-hidden>#53(ns-555.awsdns-55.com) in 4 ms

mycompany.com.  169201  IN  NS  ns-666.awsdns-66.net.
mycompany.com.  169201  IN  NS  ns-777.awsdns-77.org.
mycompany.com.  169201  IN  NS  ns-555.awsdns-55.com.
mycompany.com.  169201  IN  NS  ns-888.awsdns-88.co.uk.
;; BAD (HORIZONTAL) REFERRAL
;; Received 888 bytes from <some-ip-hidden>#53(ns-888.awsdns-88.co.uk) in 4 ms

mycompany.com.  169201  IN  NS  ns-777.awsdns-77.org.
mycompany.com.  169201  IN  NS  ns-666.awsdns-66.net.
mycompany.com.  169201  IN  NS  ns-888.awsdns-88.co.uk.
mycompany.com.  169201  IN  NS  ns-555.awsdns-55.com.
;; BAD (HORIZONTAL) REFERRAL
;; Received 888 bytes from <some-ip-hidden>#53(ns-777.awsdns-77.org) in 5 ms

mycompany.com.  169201  IN  NS  ns-777.awsdns-77.org.
mycompany.com.  169201  IN  NS  ns-888.awsdns-88.co.uk.
mycompany.com.  169201  IN  NS  ns-555.awsdns-55.com.
mycompany.com.  169201  IN  NS  ns-666.awsdns-66.net.
;; BAD (HORIZONTAL) REFERRAL

Key takeaways:

I cannot get the CNAME with any command from my terminal
The certificate is not validating

Appendix

Does this answer help you? https://stackoverflow.com/questions/57644466/missing-dns-validation-record-when-using-terraform-aws-acm-certificate-validatio — Ben Whaley, Nov 05 '21 at 22:26
Thanks @BenWhaley but I cannot seem to find a solution there. — Dimi, Nov 05 '21 at 22:32
Can you make it work manually in AWS Console? In other words, the problem is with TF code only, which is not shown by the way. — Marcin, Nov 05 '21 at 23:19
@Marcin I just rewrote the question in great detail, and no I cannot make it work in AWS Console. — Dimi, Nov 08 '21 at 09:48
One thing is not clear (to me). How does the fact you are using two accounts come in play here? I assume you run the Terraform code in our dev account. So what did you set up in the production account? Did you delegate the `sre.mycompany.com` to a zone in dev account? With limited information here - my hunch is that your problem is with DNS (not Terraform) and the split between accounts has something to do with it. — Grzegorz Oledzki, Nov 08 '21 at 21:57
Also is the difference between `ctf.sre.mycompany.com` and `challenge.sre.mycompany.com` a miss in the editing for SO or a mistake in the configuration? — Grzegorz Oledzki, Nov 08 '21 at 21:59
So you bought the domain in Route53, or you are using some other domain registar? — Marcin, Nov 09 '21 at 02:02
Hi , for the cert validation to be created you need to create first the route53 zone and add the subdomain nameservers to the domain nameservers route53 zone or to whatever provider is your domain. After that you can create an acm cert for your subdomain and can be validated with aws_acm_certificate_validation. — bembas, Nov 09 '21 at 15:49
@Marcin I am unaware as I have no access to the production account of my company's servers. — Dimi, Nov 10 '21 at 08:08
@GrzegorzOledzki `ctf.sre.mycompany.com` was a typo in the question. Corrected it, sorry for that. I did delegate the `sre.mycompany.com` to a zone in dev account and also added the respective CNAME record under it. — Dimi, Nov 10 '21 at 08:09
@Marcin Truth is I do not really understand how the hosted zones are connected with the actual domain. This is something that confuses me. — Dimi, Nov 10 '21 at 08:11
@Dimitris to connect the hosted zone with your actual domain ( i guess its in a different provider ) you need to take the Nameservers of your hosted zone mycompany.com and put them to your 3rd party provider. I guess you have done this right ? — bembas, Nov 10 '21 at 11:34
@bembas The nameservers I get by digging `mycompany.com` are different than the ones shown in the hosted zone `sre.mycompany.com`. I guess this is the problem indeed! I guess changing the NS records in the hosted zone to the ones I am getting from the dig on the domain will make it work. Is there a chance that this (changing the NS in the hosted zone `sre.mycompany.com`) would break the functionality of `mycompany.com`? I am very confused on how DNS works in the case of subdomains. — Dimi, Nov 11 '21 at 14:04
"When you create a hosted zone, Amazon Route 53 allocates a delegation set (a set of four name servers) to serve your hosted zone. Route 53 then creates a name server (NS) record inside the zone, with the same name as your hosted zone, that lists the four allocated name servers. If you change this NS record, it doesn't change the name servers that Route 53 allocated. There are use cases when you might change the NS record, such as configuring branded name servers. However, be aware that making incorrect changes to the NS record can cause your domain to become unavailable on the internet" — Dimi, Nov 11 '21 at 14:05

score 2 · Answer 1 · edited Nov 11 '21 at 01:02

2

Everything your terraform is creating is fine, however when you create a new zone in AWS you need to add the nameservers on the ROOT DNS Panel (most likely where you bought the domain mycompany.com).

You need to add the NS entry for the subdomain you want to use (the new zone you're creating)

You can reference this article https://webmasters.stackexchange.com/questions/93897/can-i-use-different-nameservers-for-different-subdomains

edited Nov 11 '21 at 01:02

Dharman

30,962
25
85
135

answered Nov 11 '21 at 00:56

Diego Velez

1,584
1
16
20

This makes total sense and its probably the solution to the problem. Before making any actual changes, I want to make sure I understand this correctly. "You need to add the NS entr(ies) for the subdomain you want to use" I believe you mean the ones automatically created on my hosting zone `sre.mycompany.com`? "add the nameservers on the ROOT DNS Panel" by root DNS panel you mean the one that has the `mycompany.com` am I getting this right? – Dimi Nov 11 '21 at 14:09
1

@Dimitris exactly, try that and let me know if it works – Diego Velez Nov 11 '21 at 15:15

score 1 · Accepted Answer · answered Nov 12 '21 at 09:06

When you have multiple route53 hosted zones for domain and subdomains you need to link them together.

This can be done by adding the subdomain Nameservers in the domain hosted zone.

You cannot break the domain hosted zone by adding records, you will only break the link with the subdomain hosted zone.

So to clarify with an example, let's say you have a domain route53 hosted zone for mycomany.gr

Record Name	Type	Value
mycomany.gr	NS	ns-xxxx.org ns-xxx.uk
sre.mycompany.com	NS	ns-yyy.org ns-yyy.uk

The first row is created when you create the route53 hosted zone. After you need to take the nameservers and add them to your domain provider. With this way you link the domain with AWS and it knows its valid.

The second row you will need to add it manually after you have create your subdomain route53 hosted zone (sre.mycompany.com). The nameservers now are the ones that route53 subdomain created for you. With this way you say to route53 this domain (mycompany.com) owns this subdomain (sre.mycompany.com).

All those stuff need to be done before and ACM certificates are created and the reason is that the ACM has a domain validation which tries to create a record to your valid domain or subdomain. If your domain or subdomain isnt linked to a valid domain the acm will throw an error.

Todd Millecam · Answer 3 · 2021-11-10T23:11:03.297

Your CNAME in your zone file has a mycompany.com on the end. That's not the normal way to do a CNAME. Should probably be:

CNAME   Simple  Record
_g938534f3gfe03832h34.challenge.sre   _89432htieh4934hw043f.tkfpekghn.acm-validations.aws.

If you add the mycompany.com into the CNAME, then the actual resolve address is _g938534f3gfe03832h34.challenge.sre.mycompany.com.mycompany.com

The only way I have found to get correct validation records into route53 through terraform looks like this:

resource "aws_route53_record" "cert-verify" {
  for_each = {
    for dvo in aws_acm_certificate.cert_name.domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  name = each.value.name
  records = [each.value.record]
  ttl = 60
  type = each.value.type
  zone_id = aws_route53_zone.zone.zone_id

}

makes for a messy state file, but it works

The bare domain is a common issue, but it does not apply to this case. — Dimi, Nov 11 '21 at 14:06

AWS ACM certificate not validating

3 Answers3