I'm inquiring about the possibility of utilizing CSF to block all bots while allowing an exception for Google bots. Google has kindly provided us with a list of whitelisted IPs at the following links:
Googlebot IPs:
- https://developers.google.com/static/search/apis/ipranges/googlebot.json
- https://developers.google.com/static/search/apis/ipranges/special-crawlers.json
- https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json
- https://www.gstatic.com/ipranges/goog.json
I've attempted to research this on various forums but haven't been able to find a solution. Most resources seem to focus on blocking specific IPs rather than implementing an exclusion for Google bots.
