4

Any robots.txt revisions applied inside VS2010 save the file with BOM which basically causes Google to reject it with a 'Syntax not understood' error. There is a related question on this but the "Save With Encoding" option isn't available for text files - even if it were, there should be a solution that just works with CTRL + S rather than having to go the advanced route just to keep BOM out.

I can't believe I'm the the only person experiencing this problem, surely there's a solution?

Community
  • 1
  • 1
Nariman
  • 6,368
  • 1
  • 35
  • 50
  • 1
    robots.txt is a simple plain text file that rarely changes. Any reason you wouldn't want to do this by hand? – John Conde Sep 13 '11 at 14:38
  • @John: agreed, it rarely changes. But mistakes can be catastrophic for SEO. You don't find it bizarre that one of the most sensitive files for a web app can't be modified in the IDE? – Nariman Sep 13 '11 at 14:44

2 Answers2

1

Looks like an easy fix for this is to have a blank line or a comment as the first line of the file: http://www.12titans.net/p/robots-txt-bom.aspx

Nariman
  • 6,368
  • 1
  • 35
  • 50
  • It worth noting that you can also use the "Advanced Save Options" in the File menu to switch to a BOM-less codepage. – Tieson T. Dec 15 '13 at 09:50
0

VS adds the BOM to UTF-8 file, but why should your robots.txt file be encoded as UTF-8 anyway?

Since it contains URLs, it should be plain ASCII or ISO-8859-1. If your site URLs contain non-ASCII characters, you have to URL-encode them appropriately. Making the file plain ANSI will save all the BOM hassle.

Also see robots.txt; What encoding?

Community
  • 1
  • 1
Eran
  • 21,632
  • 6
  • 56
  • 89