42

My website needs to send out emails with Google Apps, SendGrid and MailChimp services. Google Apps is used to receive and read incoming email to my domain.

I need to set the SPF record for my domain. The following is syntactically correct (not sure about A and MX tokens):

"v=spf1 a mx include:_spf.google.com include:servers.mcsv.net include:sendgrid.net ~all"

But if I test it with http://www.kitterman.com/getspf2.py I get

PermError SPF Permanent Error: Too many DNS lookups

Similar problem as http://www.onlineaspect.com/2009/03/20/too-many-dns-lookups-in-an-spf-record/

How can I optimize/rewrite my SPF record?

Robert
  • 559
  • 1
  • 4
  • 6
  • suggestion for improvement: the link to the kitterman hosted SPF Query tool should the following: https://www.kitterman.com/spf/validate.html – phpenthusiast Mar 14 '19 at 20:03

7 Answers7

72

So, I've never had to do this before, but based on the article you sent over, this is what I came up with.

We started with:

v=spf1 a mx include:_spf.google.com include:servers.mcsv.net include:sendgrid.net ~all

We get 10 total lookups before we throw the Too many DNS lookups error:

  2 (Initial TXT & SPF Lookups)
  2 (a & mx Lookups)
  1 (_spf.google.com)
  1 (servers.mcsv.net)
 +1 (sendgrid.net)
 -----------------
  7 Lookups

So without even following the included SPF records, we have 7 lookups.

Now, let's dive a level deeper.

1. _spf.google.com

The google SPF record evaluates to:

v=spf1 include:_netblocks.google.com include:_netblocks6.google.com ?all

Each of which resolve to the following values:

# _netblocks.google.com
v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ip4:173.194.0.0/16 ?all

# _netblocks6.google.com
v=spf1 ip6:2607:f8b0:4000::/36 ip6:2a00:1450:4000::/36 ?all

So google gives us 2 more lookups, bringing the total up to 9 Lookups.

2. servers.mcsv.net

Mailchimp is a bit of a doosey because it adds a whole 3 extra lookups:

v=spf1 include:spf1.mcsv.net include:spf2.mcsv.net include:spf.mandrillapp.com ?all

I would imagine that depending on what you're sending through Mailchimp, you might be able to remove one or two of these records (but you'll have to evaluated that yourself).

Anyway, those resolve to the following:

# spf1.mcsv.net
v=spf1 ip4:207.97.237.194/31 ip4:207.97.238.88/29 ip4:207.97.240.168/29 ip4:69.20.10.80/29 ip4:69.20.41.72/27 ip4:74.205.22.1/27 ip4:69.20.90.0/26 ?all

# spf2.mcsv.net
v=spf1 ip4:204.232.163.0/24 ip4:72.26.195.64/27 ip4:74.63.47.96/27 ip4:173.231.138.192/27 ip4:173.231.139.0/24 ip4:173.231.176.0/20 ip4:205.201.128.0/24 ?all

# spf.mandrillapp.com
v=spf1 ip4:205.201.136.0/24 ip4:205.201.137.0/24 ?all

This brings us up to a total of 12 Lookups (Which is two over the limit already).

2. sendgrid.net

SendGrid ends up being the fewest number of additional lookups for us.

v=spf1 ip4:208.115.214.0/24 ip4:74.63.202.0/24 ip4:75.126.200.128/27 ip4:75.126.253.0/24 ip4:67.228.50.32/27 ip4:174.36.80.208/28 ip4:174.36.92.96/27 ip4:69.162.98.0/24 ip4:74.63.194.0/24 ip4:74.63.234.0/24 ip4:74.63.235.0/24 include:sendgrid.biz ~all

So the only additional lookup here is sendgrid.biz, which evaluates to:

v=spf1 ip4:208.115.235.0/24 ip4:74.63.231.0/24 ip4:74.63.247.0/24 ip4:74.63.236.0/24 ip4:208.115.239.0/24 ip4:173.193.132.0/24 ip4:173.193.133.0/24 ip4:208.117.48.0/20 ip4:50.31.32.0/19 ip4:198.37.144.0/20 ~all

This brings our grand total up to 14 lookups.

So our grand total is 14 Lookups. We need to get that down to 10. I've outlined a couple of options below, you may need to use more than 1 of them to get it down.

  1. Directly include some of the redirected spf records. Now that we know which servers the spf records redirect to, you could cut out the middleman and include them directly. Note: If any of the services end up changing their SPF records, you'll have to go through the process of updating yours manually.

  2. Remove some of the services that you're using. Not sure what your use case is for having all of these services, but there's definitely some overlap that you might be able to use. For instance, SendGrid supports (1) transactional outgoing mail, (2) newsletter / marketing emails, and (3) incoming mail. So there may be some reducible redundancy.

  3. Remove the MX record if it is redundant. Depending on your setup, the MX lookup can be redundant.

Hope this helps!

Swift
  • 13,118
  • 5
  • 56
  • 80
  • About the option 3: I'm not expert about MX records. Since I use only Google Apps to receive email, can I set the SPF to take MX only from Google and not SendGrid and MailChimp? – Robert Jan 10 '13 at 16:51
  • This works, for example, but doesn't include MX (so I think I need a MX for Google) v=spf1 a include:_spf.google.com include:servers.mcsv.net include:sendgrid.net ~all – Robert Jan 10 '13 at 16:52
  • I honestly don't know if MX is required for google. You'd have to consult their documentation for details. I know you need a separate MX record for incoming from SendGrid (not sure why you would need it in the SPF lookup) – Swift Jan 10 '13 at 16:58
  • 2
    In SPF, the MX entry means to trust whichever host is designated as the MX for your domain. If you don't have a separate MX record for your domain, or if it is already covered by the other SPF rules you have, you don't need to include it. On the other hand, if you do have a designated MX for your domain which handles outgoing mail, not authorizing it to send mail would seem ... counter-productive. – tripleee Jan 15 '13 at 12:58
  • 1
    For anyone looking, [here](http://www.openspf.org/RFC_4408#mech-mx) is the relevant section of the RFC limiting MX lookups to 10. – Xiong Chiamiov Apr 10 '14 at 22:14
  • The relevant section of the RFC for DNS lookups in general is [10.1. Processing Limits](http://www.openspf.org/RFC_4408#processing-limits) – cherdt Nov 03 '14 at 16:58
  • For anyone reading this now note that RFC 4408 is now obsolete. The current RFC is 7208 and the processing limits can be found here: http://tools.ietf.org/html/rfc7208#section-4.6.4 – Steve365 Feb 17 '16 at 17:43
  • 2
    Also note the example above is slightly wrong, the "Initial TXT & SPF Lookups" are not included in the limit of 10 lookups (at least that's how I interpret the RFC...!) – Steve365 Feb 17 '16 at 17:47
  • Are `include=` counted as lookups? If they don't have some sort of `spf.example.com` subdomain specified, they may lookup a lot of TXT records. – Martijn Aug 16 '17 at 09:26
  • Hi @Swift , how do you know the extra lookups?, I mean, how do you know that _spf.google.com evaluates also _netblocks.google.com include:_netblocks6.google.com and those ones resolve the ips you mention? Thanks indeed – Jose Gallo Apr 22 '19 at 16:49
  • For others wondering how to find out exactly which lookups each `include` resolves to, I found [the SPF-checker at dmarcanalyzer.com](https://www.dmarcanalyzer.com/spf/checker) to be very useful. Run the tool on your domain, see what each `include` itself includes, and edit your DNS record to include those directly instead. I needed this to be able to add Firebase + Gmail to a Hostgator site, which requires including `websitewelcome.com`. Turns out most includes were duplicated two or more times, resulting in "Too many DNS lookups". – Magnus Sep 04 '21 at 14:43
8

Swift's answer is excellent.

A technique which is not mentioned above is to look at whether separate subdomains with their own SPF records can be used for systems that send mail via these different routes.

E.g. if the domain is example.com, have google apps send from addresses like user@gapps.example.com. Then there can be a SPF record for gapps.example.com which includes _spf.google.com, and _spf.google.com can be removed from the main example.com SPF record , which reduces lookups by 3.

Ján Sáreník
  • 1,517
  • 1
  • 16
  • 13
mc0e
  • 2,699
  • 28
  • 25
6

Have a look at SPF-tools* which help with reassembling the SPF record from the original one that uses includes into a static one containing merely ip4 and ip6 fields. It can be easily coupled together with locally-run DNS server or hosted DNS service using their API to keep everything in sync with the upstream includes.

*I am the author (now along with other contributors) and it is open source under Apache 2.0 license.

Ján Sáreník
  • 1,517
  • 1
  • 16
  • 13
0

We explored flattening the SPF records into IPs as well a creating subdomains. All of them seemed like a lot of work. We found a service from spfproxy.org which literally takes a couple minutes to setup. They basically mask the DNS lookups behind them using SPF macros. Not sure why more companies don't offer this.

AngularNerd
  • 871
  • 7
  • 14
0

A few years ago I wrote hydrate-spf, a tool that looks up includes and merges the result into one giant record. As mentioned in the README, this approach isn't ideal - it removes the ability for your included domains to update their records. However, when you're bumping up against the allowed limit, it will solve the immediate problem, and can be kept somewhat maintainable through regular updates.

Xiong Chiamiov
  • 13,076
  • 9
  • 63
  • 101
  • 1
    Need to be careful with this. Technically an SPF record longer than 255 characters is also invalid. – Mike Mar 01 '17 at 04:43
  • 1
    It really seems like we need a modern replacement for SPF that takes into account all these limitations that may have seemed reasonable at the time, but we bump into regularly nowadays. – Xiong Chiamiov Mar 26 '17 at 21:54
0

The 10 lookup limit is a limit for DNS lookups. Flattening the SPF record to include less DNS lookups and substituting them for IPs (flattening) is a way to get around the limit.

You could do this manually, but then you have to update your SPF records every time one of the providers changes their IPs (which happens frequently).

The ideal solution is to use an SPF flattening service. This one is free for low volumes, or cheap for more than 500 emails/month. It regularly polls the SPF records you want to include for updated IPs.

Fraudmarc.com

Disclosure: I am not associated with this company and this is not a referral link

Gaia
  • 2,872
  • 1
  • 41
  • 59
0

This 10-DNS-lookup limit is imposed by SPF implementations to prevent DDoS attacks against the DNS infrastructure.

With DMARCLY's Safe SPF feature, you can lift the limit without rewriting your SPF record.

lgc_ustc
  • 1,524
  • 2
  • 20
  • 31