Given that an application has:
- robots.txt contents,
- URL of interest and
- browsing entity metadata (like user-agent string, etc)
how to check if a particular URL is allowed by robots.txt?
Asked
Active
Viewed 601 times
1
Denis Kulagin
- 8,472
- 17
- 60
- 129
1 Answers
4
crawler-commons is a Java API which can parse robots files given a particular robot name and return the rules applicable for that robot. The rules have an isAllowed(String url) method which does what you are after.
Julien Nioche
- 4,772
- 1
- 22
- 28
-
2Cool. That's exactly what is was looking for. – Denis Kulagin May 29 '18 at 11:11
-
@DenisKulagin worth marking my answer as accepted then? – Julien Nioche May 29 '18 at 13:30
-
That's for sure! – Denis Kulagin May 29 '18 at 14:46