1

I have a composite c1 website. I like to use the Microsoft Search Server functionality. I have installed all the software, but when I search I get every page as a result, because the navigation menu is indexed on every page. When I use the class="noindex" option on the of the navigation menu the links used in the menu are not followed by the crawler.

Can anybody tell me how I can exclude content, but make sure that the crawler still follows the links used in the content?

matino
  • 17,199
  • 8
  • 49
  • 58
Jeroen
  • 307
  • 1
  • 6
  • 17

2 Answers2

2

Consider to have Sitemap Page with navigation structure of your site and have somewhere the link to this page which can be followed by crawler...then instead of having all pages in results when searching by some keyword from "navigation titles" you will have only the Sitemap page.

Inna
  • 361
  • 1
  • 6
1

This is basically an issue with Microsoft Search Server, but the C1 Package for Composite C1 tries to tackle this problem. Quoting from http://docs.composite.net/Composite.Search.MicrosoftSearchServer :

Excluding page content from indexing

Make sure that RemapperHttpModule is specified in the Web.config on your C1 website. If not, do the following:

  1. Open the Web.config on your C1 website.
  2. Add the following configuration under <httpModules> and <modules> (for IIS 7.0 or later) elements: <add name="MssRemaper" type="Composite.Search.MicrosoftSearchServer.RemapperHttpModule" />

Now, to exclude a part of a page from being indexed by Microsoft Search Server:

  1. Open the page in Source Editor.
  2. Locate the element you want to exclude from indexing.
  3. Add the class attribute with the "noindex" value to the element (class="noindex"): e.g. <div class="noindex">
  4. Re-index your website.

Ensuring the crawler find your pages

The exclusion of navigation elements will happen on the Composite C1 side, so doing this will remove the Search Server crawlers ability to follow those links. You can solve this by making sure you have a sitemap on your website and make Search Server index the URL to the sitemap page.

mawtex
  • 1,564
  • 1
  • 12
  • 21
  • Thanks for the answer, but the problem is, when I use the
    the hyperlinks in the div (to the other pages) are not followed by the crawler. But when I set the class="noindex" to the hyperlink, the url is followed, but the menu is also included.
     – Jeroen Nov 29 '11 at 13:37
  • Sorry - I have added Inna's tip to the answer above. +1 to @Inna :) – mawtex Nov 30 '11 at 14:45