3

Not so much time ago Google announced that now its crawler is able to index ajax-generated content on pages (you just need to follow some rules). But in my case, I need to be sure that any search engine will not be able to crawl my ajax-generated content. So, the question is:
How to prevent search engines from indexing ajax-generated content?

Thanks a bunch!

James Akwuh
  • 2,169
  • 1
  • 23
  • 25
  • 1
    If the ajax content is loaded only after a specific link was clicked, you can add `rel=nofollow`. Look [here](https://support.google.com/webmasters/answer/96569?hl=en). – DavidDomain Jul 12 '15 at 17:45

1 Answers1

1

A good start would be to read Google's documentation on this topic.

If you don't have any hashtags, then just block out this page in robots.txt as usual. Google should respect this but bear in mind that other crawlers, especially those lesser known might not.

Another idea that comes to mind is to check user-agent on your AJAX request. But then again, this doesn't prevent user-agent spoofing so there will still be a subset of rouge crawlers which are out to get your sensitive content.

You could probably find some other solution, perhaps a sort of smart JavaScript hack, which will prevent most crawlers from downloading your content but this approach will never be reliable or sustainable because ultimately, there are people committed 24/7 to making better crawlers.

If your goal is to make absolutely sure something is not indexed then it doesn't matter if it's AJAX or not. Any sensitive data needs to be hidden behind some sort of authentication or a Turing test like Captcha.

Marcin Wasilewski
  • 685
  • 1
  • 10
  • 26
  • Hi, thanks! I've also read it, and it seems there is no proper information on the question. I don't have any hashtags on my website, so I can't follow Google's advice. It's just a simple html page with ajax-generated content, which appears after page is loaded. – James Akwuh Jul 12 '15 at 17:53
  • Thanks a lot! I've found a [link](http://stackoverflow.com/questions/544450/detecting-honest-web-crawlers) to help others. – James Akwuh Jul 12 '15 at 19:31