1

We are using the robots.txt to reference our sitemap index file. Now we will release new, different countries. Our webseite under the TLD .de provides a robots.txt, containing a reference to our index file. The index files refers to different sitemaps containing our .de link in loc XML node. Other locales (eg. for .fr) are listed with xhtml:link below.

Example:

<url>
    <loc>https://xy.de/hallo</loc>
    <xhtml:link>https://xy.fr/hello</xhtml:link>
</url>

The question is now, should we add a robots.txt with a reference to our sitemap index to our .fr index too? Or might it is enough to place the reference only in the German .de robots.txt because the locations are described with alternative locations for each other locale? Or should we invert the loc XML node with the "current" locale? E.g. under https://xy.fr/robots.txt should there be a sitemap referenced with .fr links in the loc XML node?

unor
  • 92,415
  • 26
  • 211
  • 360
Tim
  • 2,236
  • 1
  • 20
  • 26

1 Answers1

0

The Sitemaps protocol doesn’t mention an xhtml:link element, so consumers following the protocol might ignore it.

As a sitemap can only contain URLs from the same host, and a robots.txt file also only works for its host, the typical way is to give each host its own robots.txt file which points to this host’s sitemap (with an absolute URL).

# robots.txt from http://fr.example/robots.txt
Sitemap: http://fr.example/sitemap.xml

# robots.txt from http://de.example/robots.txt
Sitemap: http://de.example/sitemap.xml

The sitemap can be hosted on a different host, but you still need to prove ownership via the robots.txt file (see Sitemaps & Cross Submits).

Community
  • 1
  • 1
unor
  • 92,415
  • 26
  • 211
  • 360