7

Does FastAPI need robots.txt and the tag noindex? I am creating business api app which shouldn't be called by anonymous. So I wonder whether I have to prepare robots.txt and the tag noindex in order to avoid any crawler's action or not.

I made robots.txt router as follows:

@router.get('/robots.txt')
def robots():
    data = """
    User-agent: *
    Disallow: /
    """
    return Response(content=data, media_type='text/plain')

Do I need another change?

tomo
  • 71
  • 2

1 Answers1

2

I tried this and it works

from fastapi.responses import PlainTextResponse
@app.get('/robots.txt', response_class=PlainTextResponse)
def robots():
    data = """User-agent: *\nDisallow: /"""
    return data
Welpike
  • 21
  • 2