Where to put robots.txt in tomcat 7?

Question

I'm using Tomcat 7 to host my application. I've used a ROOT.xml file under tomcat-home\conf\Catalina\localhost

<Context 
  docBase="C:\Program Files\Apache Software Foundation\Tomcat 7.0\mywebapp\MyApplication" 
  path="" 
  reloadable="true" 
/>

This is to load my webapp in the root context.

But now I'm confused as to where to put the robots.txt and sitemap.xml files. When I put in under C:\Program Files\Apache Software Foundation\Tomcat 7.0\mywebapp\MyApplication, it doesn't show up.

I've also tried placing it inside C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\ROOT , just in case. But nothing works. I'm still getting 404 not found. Can anyone please guide.

p.s. My web application runs fine. It's just that my robots.txt file is unreachable.

EDIT

My web.xml file:

    <filter>
        <filter-name>springSecurityFilterChain</filter-name>
        <filter-class>org.springframework.web.filter.DelegatingFilterProxy
        </filter-class>
    </filter>

    <filter-mapping>
        <filter-name>springSecurityFilterChain</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>

score 6 · Answer 1 · answered Apr 18 '13 at 07:52

6

Just put the robots.txt in Tomcat_DIR/webapps/ROOT/

answered Apr 18 '13 at 07:52

penguin

69
1
1

score 4 · Accepted Answer · edited Dec 15 '12 at 11:18

4

It should be in your context directory

mywebapp\MyApplication

just where you will have your web-inf directory,index.html etc.

UPDATE:

As per our discussion all urls are handled by Spring MVC DelegatingFilterProxy , so you need to exclude *.txt from this filter somehow, may be by extending DelegatingFilterProxy and configuring that filter in web.xml

SOLUTION: I can server static content through this hack in Spring MVC:

<mvc:resources mapping="/robots.txt" location="/robots.txt" order="0"/>

Saved me a ton of headache. :) Writing it for anyone else to benefit from.

edited Dec 15 '12 at 11:18

LittleLebowski

7,691
13
47
72

answered Dec 15 '12 at 10:11

Subin Sebastian

10,870
3
37
42

I've put it there, but it still gives me 404. – LittleLebowski Dec 15 '12 at 10:14
what is the link where you get 404? is it `http://serverip:port/robots.txt` ? if so `http://serverip:port/index.html` is it loading – Subin Sebastian Dec 15 '12 at 10:15
Yes, http://serverip:port loads. I've my home page file under WEB-INF/jsp/home.jsp and when I goto http://serverip:port, it loads. – LittleLebowski Dec 15 '12 at 10:18
can you post your web.xml to see if you have any special mapping – Subin Sebastian Dec 15 '12 at 10:19
Let me update the question, so that the web.xml is more viewable. – LittleLebowski Dec 15 '12 at 10:20
1

ok problem is ` springSecurityFilterChain /* ` all requests are routed through DelegatingFilterProxy – Subin Sebastian Dec 15 '12 at 10:23
1

instead of /* can you add specific types like *.html/*.htm/*.do etc, so that *.txt is not filtered – Subin Sebastian Dec 15 '12 at 10:24
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/21177/discussion-between-littlelebowski-and-subin-s) – LittleLebowski Dec 15 '12 at 10:25
its not a hack, its the normal method for serving static content – NimChimpsky Mar 04 '13 at 15:12
I followed your answer but I cannot access it yet. My question is here http://stackoverflow.com/questions/32302465/i-cannot-access-robots-txt-in-spring-mvc – Jack Aug 31 '15 at 00:46

Artem Zaika · Answer 3 · 2015-08-28T09:50:55.697

I've got the same issue: I doesn't meter where I put my robots.txt or configure <mvc:resources mapping="/robots.txt" location="/robots.txt" order="0"/> each request to robots.txt returned 404 status.

Problem here is that I use both Spring MVC and Backbone.js MVC on client side, which has its routing. So I couldn't adjust <url-pattern></url-pattern> to serve both MVCs.

The only quick and ugly solution I found here is to process requests to /robots.txt as a request in Spring MVC

@RequestMapping(value = "/robots.txt", produces = {"text/plain"}, method = RequestMethod.GET)
@ResponseBody
public String getRobotsTxt() {
    return "User-agent: *" + 
   "\n" + "Allow: /js/views/*.js" + 
   "\n" + "Allow: /js/*.js" + 
   "\n" + "Allow: /css/*.css" + 
   "\n" + "Allow: /css/*.png" + 
   "\n" + "Disallow: /cgi-bin" + 
   "\n" + "Sitemap: http://blabla/sitemap.xml";
}

Where to put robots.txt in tomcat 7?

EDIT

3 Answers3

Linked