Unwanted Crawling of pages

Question

Possible Duplicate:
How to detect fake users ( crawlers ) and cURL

some pages of my Website is getting crawled undesirably .

say pages like

abc.com/smarty/templates/1.html

abc.com/smarty/templates/2.html

abc.com/images/1.jpg

abc.com/images

I want to avoid indexing of these page
also want to remove these pages from google indexing.

I know to do this using apache setting.But since i am using shared network, i dnt hv access to these.

Please help

I really dont know why php-ers put their "core" files into webroot... — Zaffy, Dec 11 '12 at 16:51
how to make rules in robot.txt , if i want to avoid all pages begining with abc.txt/smarty to not be crawled — Priya Sehpal, Dec 11 '12 at 16:53

score 3 · Answer 1 · edited May 23 '17 at 12:14

3

You can use a text file robots.txt, that the search motors find and which tells them what pages they can and can't index in your website.

Here is a good article about how to write this file: Robots.txt: What it is, Why it’s Used and How to Write it

To remove a page from Google was discussed here

edited May 23 '17 at 12:14

Community

1
1

answered Dec 11 '12 at 16:55

Sami

648
8
25

thanx , also how to remove existing indexed pages? – Priya Sehpal Dec 11 '12 at 16:56
And how to remove directory say i dnt want abc.com/aa/ to be indexed but i want abc/aa/a.php to be indexed – Priya Sehpal Dec 11 '12 at 16:58
@PriyaSehpal just use Disallow: aa/* or a* to be more discrete. – Zaffy Dec 11 '12 at 18:02

score 0 · Answer 2 · answered Dec 11 '12 at 17:01

0

in robots.txt add rules like

User-agent:    *
Disallow:   /smarty/*
Disallow:   /images/*

answered Dec 11 '12 at 17:01

chicharito

1,047
3
12
41

score 0 · Answer 3 · answered Dec 11 '12 at 17:06

0

www.google.com/webmasters/tools/

Pic of webmaster tools!

You can fetch pages as google You can remove pages from google's index you can upload your sitemap and much more. I think this is exactly what you are looking for.

answered Dec 11 '12 at 17:06

crh225

821
18
34

Unwanted Crawling of pages

3 Answers3