Accessing robots.txt file in java

Question

I am new in java.I want to make a simple web crawler.how to access a robots.txt file for a website in java.actually i dont know much about robots.txt. plz help me out.

The robots.txt file is in a pretty standard location on every single website (since any given number of various search engines need to be able to find it). Accessing it is as simple as performing a get of [url]/robots.txt ;) — Mike McMahon, Apr 10 '12 at 23:45

score 1 · Answer 1 · edited May 23 '17 at 12:04

1

You need to solve two tasks:

use a HTTP library to fetch files over HTTP -- How to send HTTP request in java?
write or use a parser for robots.txt files -- robots.txt parser java

edited May 23 '17 at 12:04

Community

1
1

answered Apr 10 '12 at 23:52

j13r

2,576
2
21
28

Accessing robots.txt file in java

1 Answers1