Can we identify googlebot like search engines hit on particular URL

Question

My Problem:

My client site which displays more products and it adds more page load/weight. So i decided to use ajax more products loading and it works well. But here it affects the seo - and no products or deals has been indexed(Even i suggest the client to submit product via googlebase but client doesnot like that idea and he wants direct google crawling into site also he wants less time page load).

Question: Can we identify the googlebot crawling request to the server or mozila like browser user agent request to the site(server).

Suggestion I have

I tried to identify user agent from requests but that doesnot working(or i might missing something?) Please anyone have correct solution for this problem to reduce the page load time using ajax and get googlebot also to crawl the website.

Answer 1

You should just search stackoverflow for "Google AJAX SEO". There are a number of questions around this.

In short, Google has a specification to make AJAX sites crawlable: https://developers.google.com/webmasters/ajax-crawling/docs/getting-started?hl=sv-SE

You can also look into PushState as an SEO option as well.

Answer 2

One tactic that is used to solve this is to harness the pagination function of whatever framework or CMS you are using. You load one page of content and display pagination links in your view then use JavaScript to hide the pagination links and fetch the content of the linked pagination page via Ajax and append it to the current page. Take a look at how infinite-scroll works for inspiration:

http://www.infinite-scroll.com/

Basically you need to be at least loading links to pages that contain the other content so that search engines can crawl the content, but you can hide these links for the users who have JavaScript Enabled.

But to better answer your question, it is possible to redirect robots using htaccess: redirect all bots using htaccess apache

But it is better SEO, as far as I understand it, to have the content or links to it, actually available on the page.