2

I have a site which has implemented Google's shebang '#!' syntax for ajax crawling.

Both sides of the system have been implemented as discussed at https://developers.google.com/webmasters/ajax-crawling/docs/specification

In other words, URLs for an Ajax-created page such as: http://avianguitars.ca/configurator/#!S is mapped to static HTML versions which is cached server side at http://avianguitars.ca/configurator/?_escaped_fragment_=S

Google should see the #! syntax, crawl the _escaped_fragment_ version, and display it on Google with the original http://avianguitars.ca/configurator/#!S URL.

I have also added URLs in the format http://avianguitars.ca/configurator/#!S to the site's sitemap, which is submitted through Google's developer tools, however the !# URLs all appear to be ignored and are still not indexed.

Another example of a 'shebanged' URL is http://avianguitars.ca/configurator/#!action=showline&lineid=46 which maps to http://avianguitars.ca/configurator/?_escaped_fragment_=action%3Dshowline%26lineid%3D46

All other (non-ajax) URLs submitted via the sitemap are accepted, and I assume that general crawling would have picked up the URLs by now too.

Is there anything obvious I am missing which is preventing the links from being indexed?

fred2
  • 1,015
  • 2
  • 9
  • 29
  • what is configurator here? Is it a path to a folder? As i get it after visiting your site it is something like action to tell your index.php which content to load, in which case this is a problem. – guramidev Oct 26 '14 at 12:21
  • As of 2023, this is a long-obsolete way of handling ajax indexing on Google and I wouldn't recommend anybody consider it. – fred2 Apr 26 '23 at 17:17

2 Answers2

1

Yes, you have missed something. You need to add <meta name="fragment" content="!" /> to tell that your shebang URL uses an identifier fragment.

This will tell you more on the side of Google: https://developers.google.com/webmasters/ajax-crawling/docs/specification

adrenalin
  • 1,656
  • 1
  • 15
  • 25
  • I thought that that was optional - ie only required if you are not using #! fragments: "The application must opt into the AJAX crawling scheme to notify the crawler to request ugly URLs. An application can opt in with **either** or both of the following: 1. Use #! in your site's hash fragments. 2. Add a trigger to the head of the HTML of a page without a hash fragment (for example, your home page):" – fred2 Oct 25 '14 at 04:21
  • 1
    Please test it with this nevertheless. It's what you will need to have also for Facebook URL debugger to provide you the information, testable here: https://developers.facebook.com/tools/debug - funnily https://www.google.com/webmasters/tools/googlebot-fetch fails to fetch the fragmented path EVEN THOUGH it first displays the JS generated content when requesting for a rendering and secondly all the pages appear in the site search (searching with site:$domain). – adrenalin Oct 25 '14 at 09:03
  • Hi again ... this has me confused. Facebook's debug tool is still refusing to respect the #! syntax and scrape the _escaped_fragment_ urls as does Google. I tried removing the og:url meta tag and all open graph tags, as I thought that might be responsible, but seemingly not. – fred2 Oct 25 '14 at 17:05
  • 1
    I just tested on a site that I have made and FB does respect that syntax on my test, so I'd guess that it might be something else as well. – adrenalin Oct 26 '14 at 12:10
  • Thanks ... suggests I need to test with some stripped down code to see what it is that is messing up the system. Thanks for your help. – fred2 Oct 26 '14 at 15:04
  • Need to take another look, however I've awarded you the bounty as most useful, thanks again. – fred2 Oct 29 '14 at 03:44
1

Make sure to output a link rel=canonical tag in your HEAD when the page is requested using the _escaped_fragment parameter.

So when your page is requested using the special parameter, include this in your output:

<link rel="canonical" href="http://avianguitars.ca/configurator/#!action=showline&lineid=46"/>

I have a site that is completely hashbang-based and it is fully indexed (312 URLs submitted, 309 URLs indexed). I just compared your site to mine, and this is the only thing that seems to be missing. I am not sure this is the solution to your problem, but i dont see it in your html. I added this tag because i read it somewhere in the google documentation, but now i cant seem to find it anymore ...

Also, my site does not include this:

<meta name="fragment" content="!" />

... but that didnt prevent it from being indexed.

Also, the url's in google sitemap should use the hashbang syntax, NOT the _escaped_fragment syntax. If you keep having problems, try registering your website with google webmaster tools and see if any problems are reported there:

https://www.google.com/webmasters/tools/

Davy
  • 6,295
  • 5
  • 27
  • 38
  • This is great, I am going to try this too. Unfortunately the bounty has expired, but ... I don't know, I'll try to work something out if this works! – fred2 Oct 30 '14 at 18:29
  • 1
    Dont worry about that, glad if i can help. Let me know if this was indeed the case (would be weird since the google docs dont mention it) – Davy Nov 03 '14 at 08:00