0

I am having issue on robots.txt file where red dot is somehow added in the beginning of the file.

  1. I am working in asp.net c#
  2. I am creating robots.txt file on my live site
  3. and Google search console is reading the live page

In my code, I don't see any dot or space or any additional characters, but a red dot appears in search console.

I have tried to change encoding type to ASCII from UTF-8 but nothing changed with that. Please find code and screenshot of error below.

enter image description here

public ContentResult BuilRobotsText()
{
StringBuilder sb = new StringBuilder();
sb.AppendLine("User-agent: *");
sb.AppendLine("Disallow: /multimedia/");  
sb.AppendLine("Disallow: /fr-ca/media-gallery/details");
            sb.AppendLine("Disallow: /en-ca/media-gallery/details");
            sb.AppendLine("Disallow: /index.cfm");
            sb.AppendLine("Disallow: /index.cfm/");
            sb.AppendLine("Disallow: /en-ca/my-account");
            sb.AppendLine("Disallow: /fr-ca/my-account");
            sb.AppendLine("Disallow: /en-ca/error/");
            sb.AppendLine("Disallow: /fr-ca/error/");
            sb.AppendLine("Disallow: /*.html$");
            sb.AppendLine("Disallow: /*.aspx$");
            sb.AppendLine("Disallow: /*.aspx*");


var result = new ContentResult();
            // Set the content-type         
            result.ContentType = "text";
            //result.ContentEncoding = Encoding.UTF8;
            result.ContentEncoding = Encoding.ASCII;
            result.Content = sb.ToString().TrimStart();

            return result;
}

Any help would be appreciated. Thanks in advance.

Learner
  • 71
  • 1
  • 11
  • I don't see where you're actually writing the file, but maybe this is relevant: https://stackoverflow.com/questions/5266069/streamwriter-and-utf-8-byte-order-marks – itsme86 Apr 12 '19 at 14:29
  • I am not creating/writing the file. This is the text which is being returned on the page call, and robots.txt read it on url call. – Learner Apr 12 '19 at 19:26
  • "2. I am creating robots.txt file on my live site" What do you mean you're not creating/writing the file? Please explain what is actually going on in less vague terms. – itsme86 Apr 12 '19 at 19:27
  • I am getting the same red dot. But I am not generating the file. I am manually typing it in in Visual Studio... – ChrisFox Nov 13 '20 at 08:38

0 Answers0