Stack Overflow
Stack Overflow

Questions tagged [search-engine-bots]

51 questions
8
votes
4 answers

Allow search bots to crawl your sites without session IDs

Google's Webmaster guidelines state Allow search bots to crawl your sites without session IDs or arguments that track their path through the site. These techniques are useful for tracking individual user behavior, but the access pattern of bots is…
kenwarner
  • 28,650
  • 28
  • 130
  • 173
3
votes
2 answers

Are search-bots or spam-bots able to emulate/trigger JavaScript events?

Are search-bot or spam-bots able to emulate/trigger JavaScript events while they read out the page?
user798596
3
votes
1 answer

If I respond to requests for robots.txt with HTTP code 418 AKA "I'm a teapot", will this make search engines dislike me?

I have a very simple webapp that runs within HTML5's Canvas that doesn't have any public files that need to be indexed by search engines (beyond the front-page HTML file that includes calls to all the necessary resources). As such, I don't really…
thisissami
  • 15,445
  • 16
  • 47
  • 74
3
votes
2 answers

Using Javascript to get around SEO concerns

I would like to know at which stage is it okay to start manipulating HTML elements/content using Javascript so as not to impair SEO? I have read somewhere that HTML content that is hidden using the CSS property display:none is often penalized by…
magz
  • 175
  • 2
  • 8
3
votes
2 answers

How to stop all search engines, bots to crawl some urls

I want to count ads click on a widget. I've used on robot.txt file: User-agent: * Allow: / Disallow: */ads_count/* I've also add nofollow for all links in that widget. But many bots still follow urls in that widget. I've get client ip to count urls…
Vô Danh Vô Hình
  • 61
  • 5
2
votes
0 answers

How can I help search engine robots index geolocation results?

One of my sites has a few public pages that serve results based on the location the person is searching. Similar to going to a weather website and being able to search for your local weather. However for this, it is more about events in that…
RiddlerDev
  • 7,370
  • 5
  • 46
  • 62
2
votes
4 answers

Google crawler finds robots.txt, but can't download it

Can anyone tell me what's wrong with this robots.txt? http://bizup.cloudapp.net/robots.txt The following is the error I get in Google Webmaster Tools: Sitemap errors and warnings Line Status Details Errors - Network unreachable: robots.txt…
Fabio Milheiro
  • 8,100
  • 17
  • 57
  • 96
2
votes
1 answer

How to block content hotlinking except from google indexing with htaccess rules

I have prepared a .htaccess file and have placed it in a directory with pdf files to prevent hotlinking except from my site as follow: RewriteEngine On RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?example.com [NC] RewriteRule ([^/]+)\.(pdf)$…
pQB
  • 3,077
  • 3
  • 23
  • 49
2
votes
1 answer

Angular & SEO - What is the latest best practice?

I have an angular app and am now thinking about fine tuning for SEO purposes. Having had a look at a number of documents and tutorials I am pretty confused! I understand that in the past you would have to use a prerendering service. But then Google…
Jack Power
  • 303
  • 1
  • 3
  • 14
2
votes
2 answers

How to handle multiple values in meta robots?

About the meta-robots, how many values can I put? Is this valid? Or multiple meta elements?
RemRem
  • 75
  • 8
2
votes
2 answers

Search robots pressing my button...? Can I prevent that?

I have a site with content that can be "liked". Not using any api for this, its a custom like button, simply incrementing the "like" stats for the piece of content. Now, I have a views counter, on each page load, and a like button. Upon pressing the…
Louis van Tonder
  • 3,664
  • 3
  • 31
  • 62
2
votes
2 answers

How to hide Robots.txt in search engine results?

I don't want to see Robots.txt in result list of search. I'm working on an asp.net project. I found some failed solutions, one of them like below: Using "X-Robots-Tag" in header. But I want to hide a txt file so there is no "header" part in…
emy
  • 664
  • 1
  • 9
  • 22
1
vote
1 answer

Robots.txt - blocking bots from adding to cart in WooCommerce

I'm not sure how good Google's robots.txt tester is and wondering if the following example of my robots.txt for me WooCommerce site will actually do the trick for blocking bots from adding to cart and crawling cart pages, while allowing good bots…
dubpir8
  • 11
  • 2
1
vote
0 answers

If I initially hide my site using noindex, how quickly can I unhide it when wanted?

Like the question says. If my new website starts low profile, with restrictive tags such as or nosnippet or nofollow, until a certain time, how quickly can I reverse those and become findable for search…
Stilez
  • 558
  • 5
  • 14
1
vote
0 answers

Will Google search results reflect website text that I replace with a script?

I'm trying to set up web documentation for a project. I want to use variables so that, when the project name inevitably changes, I can just replace it everywhere easily. For example, I want to create HTML pages with content like this: "Get started…
Kalavinka
  • 15
  • 2
1
2 3 4