1

Google bot is crawling product filter parameters like following:

/shop/?filter_size=10

/shop/?filter_color=red

/shop/?filter_color=blue?filter_size=20

I tried to add following rules in robots.txt file but i still can see that google is still crawling those kind of urls with filters

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /wp-json/
Disallow: /cart/
Disallow: /wishlist/
Disallow: /checkout/
Disallow: /my-account/
Disallow: *?filter_color=* 
Disallow: *?filter_size=* 
Disallow: *?min_price=* 
Disallow: *?max_price=*
Disallow: /*add-to-cart=*

I am using WooCommerce & Yoast plugin.

I enabled in Yoast only indexing of pages/products/product_cat.

This is causing a high cpu loads on the server as well they are irrelevant for crawling..

How to prevent google from crawling the filters of the shop?..

Nash
  • 285
  • 2
  • 15

1 Answers1

1

so after looking around the best way to do it is with blocking the shop page from getting crawled.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: Googlebot
Disallow: /shop
Disallow: /?s=
Disallow: /search
Disallow: /wp-json
Disallow: /cart
Disallow: /wishlist
Disallow: /checkout
Disallow: /my-account
Disallow: /*?*
Nash
  • 285
  • 2
  • 15
  • I noticed that you have removed trailing slash from Disallow: /cart/ to Disallow: /cart. What is the significance of this? Please guide. Thanks – KoolPal Dec 20 '20 at 06:01
  • Why not just adding `Disallow: /*?*`? [Source](https://stackoverflow.com/a/9149936/1637178) – Jan Święcki Jul 11 '21 at 16:49