What to do about robots.txt for my CakePHP app?

Question

I have been getting the following on my live CakePHP web app:

2014-02-18 04:06:00 Error: [MissingControllerException] Controller class Robots.txtController could not be found.
Exception Attributes: array (
  'class' => 'Robots.txtController',
  'plugin' => NULL,
)
Request URL: /robots.txt

What should I do?

I am using CakePHP 2.4.2

UPDATE:

This is my robots.txt. Anything else I should add? I placed it in webroot.

User-agent: *
Disallow: /admin/

I guess your root of the vhost is not pointing to app/webroot but app/ instead. — floriank, Feb 28 '14 at 10:23
Actually I don't even have a robots.txt in my webroot. I just added one. See my updated question — Kim Stacks, Feb 28 '14 at 10:35

score 8 · Answer 1 · answered Feb 21 '15 at 11:59

8

Copy the robots.txt into the /app/webroot/ directory.

answered Feb 21 '15 at 11:59

Harald Ernst

364
4
10

score 4 · Answer 2 · answered Feb 28 '14 at 17:13

The reason you were getting the error message was because a Bot or other software was requesting the file and CakePHP couldn't find it, because it didn't exist. Now that you have created a robots.txt you should not receive error message. You can check this yourself, by going to:

http://www.example.com/robots.txt

I would probably remove /admin/, don't want to advertise where your backend is!

A simple text like the following in your robots.txt file should be sufficient, remove the reference to sitemap if you don't have one:

User-agent: *
Disallow:

Sitemap: http://www.example.com/sitemap.xml

Hope you find this helpful.

Gareth Griffiths · Answer 3 · 2014-02-28T11:06:22.460

You may have to edit your .htaccess file (or /etc/lighttpd/lighttpd.conf or similar, depending on what webserver you're using) to ignore url-rewriting for /robots.txt. You must already have your webserver set up to pass requests to CakePHP, but sadly it's passing the requests for robots.txt to CakePHP. You need to configure your webserver to not rewrite the url for the request for /robots.txt, but to read the contents of the file instead.

Edit: Bear in mind, I have no knowledge of nginx, but try something like this;

server {
    listen 80;
    client_max_body_size 5M;
    server_name example.com;
    root /var/virtual/example.com/webroot;
    include common.conf;
    include php.conf;

    if (!-e $request_filename) {
        rewrite ^/(.+)$ /index.php?url=$1 last;
        break;
    }
}

Let me show you my nginx config http://stackoverflow.com/questions/22093091/how-to-convert-www-example-com-posts-123-to-example-com-posts-123 — Kim Stacks, Feb 28 '14 at 10:54

What to do about robots.txt for my CakePHP app?

3 Answers3