0

I have been monitoring the activity on my company's website like what links are being clicked and ect. and web crawlers are going crazy on our website.

I think its a problem where we should specify a NOFOLLOW on our product listing. crawlers like MSN and GoogleBot generate about 90% of our traffic from the numbers I have. There are about 200 interactions with the website, ie. someone looks at a product or goes to the "Contact Us" page, about every 2 minutes. the crawlers just keep bouncing around and looking for things like fishing lures in the "Home Decor" section.

the crawlers also seem fixated on the search term "spin n glow" we used fishing as a descriptive tag. But it has searched for "spin n glow" over 3.6 million times!

Basically, im asking is that normal? literally there is more then 2 hits a second on average. There is always a crawler indexing the page and its very hard to tell how many people are actually viewing the website.

--edit--

here is the keyword meta we use

<meta name="keywords" content="online Fishing gear, discount hunting supplies, camping gear online, hunting gear online, camping accassoires, hunting accessories, fishing accessories, Hunting, marine electronics/products, clothing, cooking, gifts, lures, tackle, live bait, guns, rifles, shotguns, Saginaw Bay, Michigan, Rivers, streams, woods, up north, sonar, rods, reels, waders, fly fishing, accessories, stream fishing, fishing reports, ice fishing, still fishing, walleye, perch, salmon, sonar, VHF radios, speed indicators, mounting systems, marine batteries, GPS, Temperature probes, UHF radios, autopilots, smokers, cooking aides, jerky cure, gas stoves, camping equipment, equipment, cook books, cookbooks, Terry Redlin, Terry Redlin Items, living stone figurines, second nature figurines, collector plates, framed pictures, birdhouses, chimes, Camille Beckman lotion, Big Sky Carvers art, Heritage Decoy, decoys, duck hunting, Spoontique lighthouses, Westland wildlife, sandstone coasters, bedding accessories, Leanin Tree cards, trout vests, fishing jackets, Red Ball, LaCrosse, Kobuk, hip boots, Columbia, Columbia Gnat Creek Jacket, Hodgman, Filson, Masterbuilt, Luhr Jensen, Camp Chef, Shore Lunch, smoking chips, smoking chunks, seasoning, wild game, bullets, ammunition, scopes, buck shot, jigs, line, fishing line" />
Grayson Briggs
  • 69
  • 1
  • 11
  • About your [previous question](http://stackoverflow.com/questions/26332296/trouble-with-string-split-java), it was closed but with wrong duplicate. It should be closed as duplicate of one of [this questions](http://stackoverflow.com/search?q=%5Bjava%5D+split+on+dot) like for instance this one http://stackoverflow.com/questions/14833008/java-string-split-with-dot. – Pshemo Oct 13 '14 at 03:12
  • In short `split` takes regex as argument and in regex `.` represents any character. To make it simple dot literal you need to escape it. – Pshemo Oct 13 '14 at 03:13

2 Answers2

1

Algorithms always change, but if you were to provide some of your META TAGS that may assist in answering. The amount of hits seems a bit high but sometimes when a specific popular items is being trended more crawls take place in that subject matter. So "spin n glow" could be one of them.

Jesse
  • 118
  • 5
0

You can also take a look at visitors' IP: if most of them have one or several IP addresses - this is a bot. Otherwise, it looks like they are real people.

Oleg
  • 823
  • 6
  • 5