robots.txt parser java

Question

I want to know how to parse the robots.txt in java.

Is there already any code?

score 5 · Answer 1 · answered Jun 29 '10 at 13:32

5

Heritrix is an open-source web crawler written in Java. Looking through their javadoc, I see that they have a utility class Robotstxt for parsing the robots.txt file.

answered Jun 29 '10 at 13:32

Bill the Lizard

398,270
210
566
880

There is a bug in Robotstxt. Please do not use it. Wasted a lot of time. To a file like this : User-agent : * Disallow: / AllowAll method of Robotstxt says "true". – 10101010 Apr 27 '15 at 06:17

score 2 · Answer 2 · answered Oct 26 '10 at 16:19

2

There's also jrobotx library hosted at SourceForge.

(Full disclosure: I spun off the code that forms that library.)

answered Oct 26 '10 at 16:19

Alan Krueger

4,701
4
35
48

score 0 · Answer 3 · edited Sep 06 '16 at 09:49

0

There is also a new release of crawler-commons:

https://github.com/crawler-commons/crawler-commons

The library aims to implement functionality common to any web crawler and this includes a very handy robots.txt parser

edited Sep 06 '16 at 09:49

Julien Nioche

4,772
1
22
28

answered Mar 14 '13 at 12:22

anastluc

301
2
4

robots.txt parser java

3 Answers3

Linked