Block Bots from crawling one of my sites on a multistore multidomain prestashop

Question

Hello i have a multistore multidomain prestashop installation with main domain example.com and i want to block all bots from crawling a subdomain site subdomain.example.com made for resellers where they can buy at lower prices because the content is duplicate to the original site, and i am not exacly sure how to do it. Usualy if i want to block the bots for a site i would use

User-agent: *
Disallow: /

But how do i use it without hurting the whole store ? and is it possible to block the bots from the htacces too ?

How do your resellers feel about this? And if the content is identical (no value added) then what's even the point in having them? — Lightness Races in Orbit, May 14 '14 at 08:34
the shop is identical as the main one however the prices are lowered accordingly as it should be for resselers — Ivan Veselinov Nedelchev, May 14 '14 at 08:35
You just repeated the original post instead of answering my question.... — Lightness Races in Orbit, May 14 '14 at 08:59
From what i know the point is some accounting stuff but that's what my employer wanted me to do so that's what i did, anyway my original question still stands — Ivan Veselinov Nedelchev, May 14 '14 at 10:54
Isn't it better to create extra groups for resellers which makes it possible to show them lower prices? — Pawel Gumiela, Aug 21 '15 at 15:34

score 1 · Answer 1 · edited May 23 '17 at 10:29

Regarding your first question: If you don't want search engines to gain access to the subdomain (sub.example.com/robots.txt), using a robots.txt file ON the subdomain is the way to go. Don't put it on your regular domain (example.com/robots.txt) - see Robots.txt reference guide.

Additionally, I would verify both domains in Google Search Console. There you can monitor and control the indexation of the subdomain and main domain.

Regarding your second question: I've found a SO thread here which explains what you want to know: Block all bots/crawlers/spiders for a special directory with htaccess.

score 0 · Answer 2 · answered Jun 12 '16 at 18:43

We use a canonical URL to tell the search engines where to find the original content.

https://yoast.com/rel-canonical/

A canonical URL allows you to tell search engines that certain similar URLs are actually one and the same. Sometimes you have products or content that is accessible under multiple URLs, or even on multiple websites. Using a canonical URL (an HTML link tag with attribute rel=canonical) these can exist without harming your rankings.

Block Bots from crawling one of my sites on a multistore multidomain prestashop

2 Answers2