1

I want to have a simple little Flask website that returns a robots.txt. Here's what I have so far:

from flask import Flask
app = Flask(__name__)
app.run(host = "0.0.0.0", port = 80)

@app.route("/robots.txt")
def robots_dot_txt():
    return "User-agent: *\nDisallow: /"

When I try to access this, I get a 404:

213.152.161.130 - -[31/Mar/2018 21:35:01] "GET /robots.txt HTTP/1.1" 404 -
BlandCorporation
  • 1,324
  • 1
  • 15
  • 33
  • Per https://stackoverflow.com/a/20648053/3990806 Just put robots.txt in your static directory. – Adam Mar 31 '18 at 20:05

2 Answers2

5

it's better to serve your static assets from statics and using the web server,But for your issue, you have to launch the "server" after you defined the routes ... so if you move your "run()" at the end of the file, you should be good.

and it's often recommended to inclose it in a if name

from flask import Flask
app = Flask(__name__)

@app.route("/robots.txt")
def robots_dot_txt():
    return "User-agent: *\nDisallow: /"

if __name__ == '__main__':
    app.run(host = "0.0.0.0", port = 80)
stamaimer
  • 6,227
  • 5
  • 34
  • 55
eMerzh
  • 485
  • 1
  • 4
  • 12
  • This works for me, except the `\n` which is not rendered. I had to use
    instead.     – tvb Nov 11 '22 at 08:03
0

Just put the robots.txt file inside the static folder in your project directory. And then set the static_url_path when you create your Flask app instance.

app
    --__init__.py
    --static
        --robots.txt
    --templates

In app.__init__.py:

from flask import Flask

app = Flask(__name__, static_url_path='')

if __name__ == "__main__":
    app.run()

Or you can use send_from_directory instead of change the default value of static_url_path:

In app.__init__.py:

from flask import Flask, send_from_directory

app = Flask(__name__)

@app.route("/robots.txt")
def robots():
    return send_from_directory("static", "robots.txt")

if __name__ == "__main__":
    app.run()
stamaimer
  • 6,227
  • 5
  • 34
  • 55