3

I got a problem with my site and the robots.txt file. I got Disallow for /feed and /trackback url on my robots, but for some reason im still receiving on my Google Webmaster url like http://www.mydomain.net/year/month/post.html/feed and http://www.mydomain.net/year/month/post.html/trackback and I not have idea how to fixe properly. After that happend my site go down on visits.

Can anyone help me, please?

Regards

This is my robots.txt

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Allow: /wp-content/uploads/
Disallow: /trackback
Disallow: /?ref=
Disallow: /feed
Disallow: /comments/feed
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$
Disallow: /?s= 
Disallow: /search

User-agent: Googlebot-Image
Allow: /

Sitemap: http://www.mydomain.net/sitemap.xml

user2143527
  • 29
  • 1
  • 7

2 Answers2

1

your robtos file seems to be ok, I think you have misunderstood how to use it. You can't prevent indexing of a certain folder by adding a regexp into a robot file, this is because the robots.txt controls the crawler activity only. So in order to prevent the indexing of certain pages you have to use the robots meta tag.

user3310736
  • 11
  • 1
0

I think your problem is the use the wildcard character * This is not required just use 

Disallow: /feed/

Same applies to all other uses of the wild character.

Also wild characters are not accepted by all search engines.

You could try one of the robots.txt syntax checkers for more assistance.

Sirko
  • 72,589
  • 19
  • 149
  • 183
Roy
  • 1
  • 1
    I'm using this and still getting 150 /feed/ errors in webmasters. Perhaps the wildcard is needed to show all URL ending in /feed/ ? – Ben Racicot Aug 23 '14 at 23:12