How do I stop search engine crawlers from indexing a Lemoon site?

Question

We've got a public site and a test site created in Lemoon 4.5.1, we'd like the test site to not be indexed at all. Is it possible and how do we do it?

We're using web deployment packages when updating the site. So if we add a robots.txt in the test site it will be overwritten everytime we deploy.

When editing a page, there is a "Meta Robots - NoIndex" setting, which probably would suit us fine, but we'd like to avoid editing every page.

If you don’t want to rely on all robots honoring a `robots.txt` – then password protect the test site using HTTP Auth. — CBroe, May 20 '14 at 08:52
Well, robots.txt would be perfect, but I don't want it to be overwritten when installing updates using a web deployment packet. — Carl Björknäs, May 20 '14 at 08:58

score 0 · Accepted Answer · answered May 26 '14 at 14:15

I solved it by adding a Meta data key (AvoidSearchEngineIndexing) on the site. In the master page that every page uses I check if it's set or not and adjusts the content in the robots meta tag.

Code behind

protected void Page_Load(object sender, EventArgs e)
{
    MetaRobotsContent = "index, follow";
    object avoidSearchEngineIndexing;
    if (Site.MetaData.TryGetValue("AvoidSearchEngineIndexing", out avoidSearchEngineIndexing) && ((avoidSearchEngineIndexing as bool?) ?? false))
    {
        MetaRobotsContent = "noindex, nofollow";
    }
}

protected string MetaRobotsContent { get; set; }

Markup

<meta name="robots" content="<%= MetaRobotsContent %>

How do I stop search engine crawlers from indexing a Lemoon site?

1 Answers1