Should I include mobile site URLs in robots.txt?

Question

My boss is having me look at various ways to improve our site's SEO and I've been doing some research on it. I'm aware that search engines like mobile-friendly sites and I used Google's Webmaster Tools, finding that it considers our site to be mobile-friendly. However, we lack an adequate robots.txt file.

What we want to do is avoid getting the same page indexed twice (as desktop and mobile versions), and he recommended that I include our site's mobile URLs in the robots.txt file. However, will doing this damage our site's ranking? I get that files listed under robots.txt shouldn't be indexed, which raises concerns about whether or not people will be able to see results for our site when they search for it on their phones.

Answer 1

Although I would not recommend having two different files or URLs for mobile/regular sites,as the official google blog recommends:

Sites that use responsive web design, i.e. sites that serve all devices on the same set of URLs, with each URL serving the same HTML to all devices and using just CSS to change how the page is rendered on the device. This is Google’s recommended configuration.

http://googlewebmastercentral.blogspot.ca/2012/06/recommendations-for-building-smartphone.html

Having said that, since you already have mobile versions and would like to block google bot from indexing multiple versions of the same URL:

Blocking Googlebot-Mobile from desktop site

Desktop site: http://www.domain.com/robots.txt

    User-agent: Googlebot
    User-agent: Slurp
    User-agent: bingbot
    Allow: /

    User-agent: Googlebot-Mobile
    User-Agent: YahooSeeker/M1A1-R2D2
    User-Agent: MSNBOT_Mobile
    Disallow: /

Mobile site: http://m.domain.com/robots.txt

    User-agent: Googlebot
    User-agent: Slurp
    User-agent: bingbot
    Disallow: /

    User-agent: Googlebot-Mobile
    User-Agent: YahooSeeker/M1A1-R2D2
    User-Agent: MSNBOT_Mobile
    Allow: /

http://searchengineland.com/5-tips-for-optimal-mobile-site-indexing-107088

Answer 2

Robots.txt disallows crawling, not indexing.

So if you would block your mobile URLs, bots would never be able to even see that you have a mobile site, which is probably not what you want.

Alternative

Tell bots what the links are about. Based on this declaration, bots can decide what they want to do with these URLs.

You can do this by providing the link types alternate and canonical:

• alternate (defined in the HTML5 spec), to denote that it’s an "alternate representation of the current document".
• canonical (defined in RFC 6596), to denote that the pages are the same, or that they only have trivial differences (e.g., different HTML structure, table sorted differently etc.), or that one is the superset of the other.

So if you want to use the URLs from the desktop site as canonical, you would use "alternate canonical" to link from mobile to desktop, and "alternate" to link from desktop to mobile. You can see an example in my answer to the Webmasters question Linking desktop and mobile pages.