16

I submitted my robots.txt file ages ago to Google and it is still giving me a syntax not understood for the first line.

After Googling the most common problem is Google adding a '?' at the start of the line but it isnt doing that to me.

the url to the robots.txt is

www.leisurepursuits.co.uk/robots.txt

The error is:

Line 1: User-agent: * Syntax not understood

Lex Eichner
  • 1,056
  • 3
  • 10
  • 35

1 Answers1

24

This error shows because the expected robots.txt file format is plain text encoded in UTF-8. The file consists of records (lines) separated by CR, CR/LF or LF.

If a character encoding is not a subset of UTF-8, then your robots.txt files is being parsed incorrectly.

First check your robots.txt url in http://www.asymptoticdesign.co.uk/cgi-bin/check-url.pl by selecting the second option “view source” and see that it responds successfully or not.

Now upload your robots.txt file again. It will not show error as shown previously.

Jessica Eldridge
  • 800
  • 6
  • 7
  • 5
    Links no longer works. http://www.milevis.com/Tips/Details/42 fixes for me – ysrb May 07 '14 at 16:48
  • 2
    I originally created the robots.txt file through VS2013, which was my issue. After reading this post, I opened in Notepad, saved, and then uploaded again and it worked. – JoshYates1980 Apr 07 '16 at 19:02
  • 3
    Throwing this in here but you can right-click on the text file in solution explorer, "open with" > "source code text with encoding" > choose "western european (windows)" and whala! You will see the extraneous characters in the beginning. – Jason Roner Mar 17 '18 at 17:07