4

We have multiple websites served from the same Sitecore instance and same production web server. Each website has its own primary and Google-news sitemap, and up to now we have included a sitemap specification for each in the .NET site's single robots.txt file.

Our SEO expert has raised the presence of different domains in the same robots.txt as a possible issue, and I can't find any documentation definitely stating one way or the other. Thank you.

Iucounu
  • 1,630
  • 1
  • 20
  • 31
  • 1
    Please check this question I think it helps you: http://stackoverflow.com/questions/19707954/implementing-sitecore-multisite-robots-txt-files/19708405#19708405 –  Jan 15 '14 at 21:23
  • I greatly appreciate the link. I still can't find documentation on whether this is actually necessary. :) – Iucounu Jan 15 '14 at 21:27
  • 1
    I am not SEO experpert but I know,it is better robots.txt to inform the search engines of sitemaps for it's own domain. –  Jan 15 '14 at 21:36
  • This question has been asked in a variety of ways and the answers are all pretty similar. Here's another non-Sitecore example: http://stackoverflow.com/questions/17033779/robots-txt-file-for-different-domains-of-same-site – Martin Davies Jan 15 '14 at 21:50
  • those answers are actually not correct for my question, which is not about how to serve different robots.txt files, but about whether it is valid or invalid to list sitemaps for different domains in the same robots.txt file. – Iucounu Jan 15 '14 at 22:07
  • 1
    I just realised I pointed you to sitemaps.xml, it's been a long day! Anyway: http://stackoverflow.com/questions/11373376/robots-txt-for-multiple-domains – jammykam Jan 15 '14 at 22:42

2 Answers2

4

This should be OK for Google at least. It may not work for other search engines such as Bing, however.

According to https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt:

sitemap: [absoluteURL]

[absoluteURL] points to a Sitemap, Sitemap Index file or equivalent URL. The URL does not have to be on the same host as the robots.txt file. Multiple sitemap entries may exist. As non-group-member records, these are not tied to any specific user-agents and may be followed by all crawlers, provided it is not disallowed.

Community
  • 1
  • 1
plasticinsect
  • 1,702
  • 1
  • 13
  • 23
1

The best way to achieve this is to Handle the Robots.txt from Sitecore Content Tree.

We also have similar structure where we are delivering multiple websites from Single sitecore instance.

I have written a blog for such please find it below. It is exactly what you want.

http://darjimaulik.wordpress.com/2013/03/06/how-to-create-handler-in-sitecore/

Maulik Darji
  • 131
  • 5