Limit access to ajax calls and block crawlers and spiders

Question

was wondering how to code a server file for ajax calls ?,

example:

Class cars, have 2 functions

reply with all brands

return $database->Listall('brands','cardb'); I call it with api.php?mod=list
reply with all cars in brand return $database->Listall('cars','cardb',"WHERE brand=$brand"); I call it with api.php?mod=list&brand=kia

the problem is that google list these api links in its search directory, + users can also access this page on there own (not throu ajax call) so how can i block this ?

score 1 · Answer 1 · answered Jun 09 '11 at 15:26

Use robots.txt to block search engine spiders. Some robots ignore this, but this is a easily way to get it out of search engines.
If user can access it in the browser, they can access it directly. There are some ways you could block it, but i doubt it would be reliable.

score 1 · Accepted Answer · answered Jun 09 '11 at 15:29

AJAX is just a way to request a page that will return simplified (xml?) content.

Block Google
Use this on your robots.txt and serve your ajax requests from the ajax folder.

User-agent: *
Disallow: /ajax-folder/

Use _POST on ajax requests and send a "secret"
Have your AJAX requests send along a "secret" variable and, on requests that don't have that variable just re-direct them to another page.

score 0 · Answer 3 · answered Jun 09 '11 at 15:27

0

Restrinct Google's access to api.php in robots.txt

answered Jun 09 '11 at 15:27

CristiC

22,068
12
57
89

score 0 · Answer 4 · answered Jun 09 '11 at 15:27

0

Simply use a token when generating the AJAX call. This if the token is not there, don't serve it.

answered Jun 09 '11 at 15:27

JohnP

49,507
13
108
140

Do you have a link that explains how to achieve this? I need it. – Michael Rogers Mar 07 '17 at 01:38
@MichaelRogers if you're looking to block crawlers then the way to go would be via the robots txt file. This would be a good place to start - https://moz.com/learn/seo/robotstxt If you're talking about a token approach, I don't really have an article for it. This question would be good place to start - http://stackoverflow.com/questions/22063612/adding-csrftoken-to-ajax-request Add the token and then check for its presence on the API end – JohnP Mar 07 '17 at 09:33

Limit access to ajax calls and block crawlers and spiders

4 Answers4