15

Please can someone tell me a simple way to find href and src tags in an html file using regular expressions in Java?
And then, how do I get the URL associated with the tag?

Thanks for any suggestion.

Michael Myers
  • 188,989
  • 46
  • 291
  • 292
Ricardo Felgueiras
  • 3,589
  • 6
  • 25
  • 27

7 Answers7

55

Using regular expressions to pull values from HTML is always a mistake. HTML syntax is a lot more complex that it may first appear and it's very easy for a page to catch out even a very complex regular expression.

Use an HTML Parser instead. See also What are the pros and cons of the leading Java HTML parsers?

Community
  • 1
  • 1
David Webb
  • 190,537
  • 57
  • 313
  • 299
  • 4
    It depends on what you are doing. If you are processing a lot of HTML from random sources an HTML Parser may well fail on some of them and will likely require more memory and processing than a regex. For example the Heritrix web crawler uses regex for link extraction on HTML pages. – Kris Mar 24 '09 at 12:19
  • 1
    Please answer the original question first and then suggest how to optimize. Many people visit this question on SO hoping to learn how to parse HTML using regular expressions, but instead find something they weren't looking for. Using regular expressions is quick and dirty and you do not have to download a separate library for it to work. – Drupad Panchal Jul 29 '11 at 19:26
  • 2
    I disagree with this answer, it is by no means always a mistake to use regex on html - as @Kris pointed out: Trying to parse a full html document often requires valid html which is not always given. And it provides a huge overkill in cases where you have a clearly defined case like finding an tag's href attribute value. – Bachi Dec 18 '13 at 10:19
21

The other answers are true. Java Regex API is not a proper tool to achieve your goal. Use efficient, secure and well tested high-level tools mentioned in the other answers.

If your question concerns rather Regex API than a real-life problem (learning purposes for example) - you can do it with the following code:

String html = "foo <a href='link1'>bar</a> baz <a href='link2'>qux</a> foo";
Pattern p = Pattern.compile("<a href='(.*?)'>");
Matcher m = p.matcher(html);
while(m.find()) {
   System.out.println(m.group(0));
   System.out.println(m.group(1));
}

And the output is:

<a href='link1'>
link1
<a href='link2'>
link2

Please note that lazy/reluctant qualifier *? must be used in order to reduce the grouping to the single tag. Group 0 is the entire match, group 1 is the next group match (next pair of parenthesis).

Henryk Konsek
  • 9,016
  • 5
  • 32
  • 41
  • Thanks. While not a real "works-everywhere" regex this works for data returned from google hot trends and I have been pulling my hair to parse it for a long time... – rjha94 Oct 17 '10 at 16:01
7

Dont use regular expressions use NekoHTML or TagSoup which are a bridge providing a SAX or DOM as in XML approach to visiting a HTML document.

mP.
  • 18,002
  • 10
  • 71
  • 105
5

If you want to go down the html parsing route, which Dave and I recommend here's the code to parse a String Data for anchor tags and print their href.

since your just using anchor tags you should be ok with just regex but if you want to do more go with a parser. The Mozilla HTML Parser is the best out there.

File parserLibraryFile = new File("lib/MozillaHtmlParser/native/bin/MozillaParser" + EnviromentController.getSharedLibraryExtension());
                String parserLibrary = parserLibraryFile.getAbsolutePath();
                //  mozilla.dist.bin directory :
                final File mozillaDistBinDirectory = new File("lib/MozillaHtmlParser/mozilla.dist.bin."+ EnviromentController.getOperatingSystemName());

        MozillaParser.init(parserLibrary,mozillaDistBinDirectory.getAbsolutePath());
MozillaParser parser = new MozillaParser();
Document domDocument = parser.parse(data);
NodeList list = domDocument.getElementsByTagName("a");

for (int i = 0; i < list.getLength(); i++) {
    Node n = list.item(i);
    NamedNodeMap m = n.getAttributes();
    if (m != null) {
        Node attrNode = m.getNamedItem("href");
        if (attrNode != null)
           System.out.println(attrNode.getNodeValue());
Scott Cowan
  • 2,652
  • 7
  • 29
  • 45
3

I searched the Regular Expression Library (http://regexlib.com/Search.aspx?k=href and http://regexlib.com/Search.aspx?k=src)

The best I found was 

((?<html>(href|src)\s*=\s*")|(?<css>url\())(?<url>.*?)(?(html)"|\))

Check out these links for more expressions:

http://regexlib.com/REDetails.aspx?regexp_id=2261

http://regexlib.com/REDetails.aspx?regexp_id=758

http://regexlib.com/REDetails.aspx?regexp_id=774

http://regexlib.com/REDetails.aspx?regexp_id=1437

Mark
  • 371
  • 1
  • 5
  • 14
  • 2
    I hate that site. I see they still don't bother to mention which flavor a given regex is targeted at. This regex (id=2261) uses named captures and conditionals, neither of which is supported by Java. – Alan Moore Mar 24 '09 at 17:03
2

Regular expressions can only parse regular languages, that's why they are called regular expressions. HTML is not a regular language, ergo it cannot be parsed by regular expressions.

HTML parsers, on the other hand, can parse HTML, that's why they are called HTML parsers.

You should use you favorite HTML parser instead.

Jörg W Mittag
  • 363,080
  • 75
  • 446
  • 653
1

Contrary to popular opinion, regular expressions are useful tools to extract data from unstructured text (which HTML is).

If you are doing complex HTML data extraction (say, find all paragraphs in a page) then HTML parsing is probably the way to go. But if you just need to get some URLs from HREFs, then a regular expression would work fine and it will be very hard to break it.

Try something like this:

/<a[^>]+href=["']?([^'"> ]+)["']?[^>]*>/i
Guss
  • 30,470
  • 17
  • 104
  • 128