How do I serve robots.txt in Spring framework?

Question

I saw this but it's obsolete. I tried the following:

Created src/main/resources/static/robots.txt.

@Configuration
public class WebConfig extends WebMvcConfigurerAdapter{

    @Override
    public void addResourceHandlers(ResourceHandlerRegistry registry) {
//        registry.addResourceHandler("/robots.txt").addResourceLocations("classpath:/static/robots.txt");
//        registry.addResourceHandler("/robots.txt").addResourceLocations("/static/robots.txt");
//        registry.addResourceHandler("/robots.txt").addResourceLocations("/static/");
        registry.addResourceHandler("/robots.txt").addResourceLocations("classpath:/static/");

Each time,

$ curl -i http://localhost:8080/robots.txt
HTTP/1.1 302
Set-Cookie: JSESSIONID=D1E5304FE8E693115A1FFB0F76786ABD; Path=/; HttpOnly
Location: http://localhost:8080/pageNotFound
Content-Length: 0

However static CSS works.

$ curl -iq http://localhost:8080/css/style.css | head
HTTP/1.1 200

I killed the server and ran mvn spring-boot:run each time I changed the code.

It says this when I start the server:

2018-05-03 17:22:18.639  INFO 25148 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/robots.txt] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
...
2018-05-03 17:22:39.139  INFO 25148 --- [nio-8080-exec-1] c.s.shorturl.apis.ShortUrlApiController  : Method name: doUrlRequest() request http method is GET

It is executing this instead.

@Controller
@Scope("session")
@ApiIgnore
public class ProjectShortUrlController implements ErrorController{
    @RequestMapping(value = "/*")
    public void doUrlRequest(HttpServletRequest request, HttpServletResponse response) {
      CustomLogger.info(TAG, "rediection: ", "Method name: doUrlRequest() request http method is "+request.getMethod());

Documentation: https://docs.spring.io/spring/docs/5.1.0.BUILD-SNAPSHOT/spring-framework-reference/web.html#mvc-config-static-resources

Spring 4.3.10

I found if I do this

@Configuration
public class WebConfig extends WebMvcConfigurationSupport {
    @Override
    public void addResourceHandlers(ResourceHandlerRegistry registry) {
registry.addResourceHandler("/robots.txt").addResourceLocations("/static/robots.txt");

Then robots.txt works

$ curl -i http://localhost:8080/robots.txt
HTTP/1.1 200
Content-Type: text/plain
Content-Length: 28

User-agent: *
Disallow: /

But all other URLs start to give "Whitelabel Error Page"!

score 3 · Accepted Answer · answered Aug 27 '18 at 15:19

I gave up on WebMvcConfigurerAdapter. I think controller requests have priority over resource handlers. I just used this in the controller.

@RequestMapping(value = "/robots.txt")
public void robots(HttpServletRequest request, HttpServletResponse response) {
    try {
        response.getWriter().write("User-agent: *\nDisallow: /\n");
    } catch (IOException e) {
        CustomLogger.info(TAG, "robots", "robots(): "+e.getMessage());
    }
}

I might have had trouble because there was a @RequestMapping(value = "/*") path.

score 0 · Answer 2 · answered Nov 12 '19 at 05:01

I tired the following way in application-context.xml, and it worked absolutely fine.

<mvc:resources mapping="/robots.txt" location="/custom/bots/robots.txt" />
<mvc:resources mapping="/favicon.ico" location="/custom/images/main_logo_pink.png" />

I have put just underneath, like the other static files in my project.

<mvc:annotation-driven />

<mvc:resources mapping="/custom/styles/**" location="/custom/styles/" />
<mvc:resources mapping="/custom/scripts/**" location="/custom/scripts/" />
<mvc:resources mapping="/custom/images/**" location="/custom/images/" />

How do I serve robots.txt in Spring framework?

2 Answers2