Retrieving URL's from webpage in java

Question

I have the most basic java code to do a http request and it works fine. I request data and a ton of html comes back. I want to retrieve all the url's from that page and list them. For a simple first test i made it look like this:

int b = line.indexOf("http://",lastE);
int e = line.indexOf("\"", b);

This works but as you can imagine it's horrible and only works in 80% of the cases. The only alternative i could come up with myself sounded slow and stupid. So my question is pretty mutch do i go from

String html

to

List<Url>

?

Same question here: http://stackoverflow.com/questions/2717760/finding-links-on-a-webpage-with-java — palacsint, Sep 13 '11 at 07:02

score 1 · Answer 1 · answered Sep 13 '11 at 06:50

1

I would try a library like HTML Parser to parse the html string and extract all url tags from it.

answered Sep 13 '11 at 06:50

flash

6,730
7
46
70

score 1 · Accepted Answer · answered Sep 13 '11 at 06:56

1

Pattern p = Pattern.compile("http://[\w^\"]++");
Matcher m = p.matcher(yourFetchedHtmlString);
while (m.find()) {
   nextUrl=m.group();//Do whatever you want with it
}

You may also have to tweak the regexp, as i have just written it without testing. This should be a very fast way to fetch urls.

answered Sep 13 '11 at 06:56

Ibolit

9,218
7
52
96

That gives me the invalid escape sequence exception – TFennis Sep 13 '11 at 06:58

score 0 · Answer 3 · answered Sep 13 '11 at 06:55

0

Your thinking is good, you just missing some parts.

Yous should add some known extension for urls. like .html .aspx .php .htm .cgi .js .pl .asp

And if you like images too then add .gif .jpg .png

I think your doing it the best just need to add more extensions checking.

If you can post the full method code, i will be happy to help you make it better.

answered Sep 13 '11 at 06:55

Danpe

18,668
21
96
131

The problem is that a lot of links don't end in an extension like index.php?page=here – TFennis Sep 13 '11 at 06:59

Retrieving URL's from webpage in java

3 Answers3