How to Extract data from HTML page using regex pattern in java

Question

I'm trying to extract data from an html page as to store them in a String array

In the HTML page values are shown like this

 <tbody>
                      <tr>
                        <td style="width: 14%;">88055</td>
                        <td style="width: 19%;" class="gris">Ville</td>
                        <td style="width: 33%;"><a href="repertoire-des-municipalites/fiche/municipalite/88055/" >Amos</a></td>
                        <td style="width: 34%;"><a href="repertoire-des-municipalites/fiche/mrc/880/" >Abitibi</a></td>
                      </tr>
                      <tr>
                        <td style="width: 14%;">85080</td>
                        <td style="width: 19%;" class="gris">Village</td>
                        <td style="width: 33%;"><a href="repertoire-des-municipalites/fiche/municipalite/85080/" >Angliers</a></td>
                        <td style="width: 34%;"><a href="repertoire-des-municipalites/fiche/mrc/850/" >Témiscamingue</a></td>
                      </tr>
                      <tr>
                        <td style="width: 14%;">87050</td>
                        <td style="width: 19%;" class="gris">Municipalité</td>
                        <td style="width: 33%;"><a href="repertoire-des-municipalites/fiche/municipalite/87050/" >Authier</a></td>
                        <td style="width: 34%;"><a href="repertoire-des-municipalites/fiche/mrc/870/" >Abitibi-Ouest</a></td>
                      </tr>

I need to extract only the string where the href = Municipality

witch means Amos ,Angliers , etc... and store them into an array of string

So far I have tried this and I'm lost

  public static final String EXPRESSION = "";//How to write the regex expression?
String [] data = new String [20]
    URL url = new URL("http://myur.com");


 BufferedReader in  = new BufferedReader(new InputStreamReader(url.openStream()));

        while ((ligne = in.readLine()) != null) {
          //What to write here? 
            }
            in.close();

P.S : I'm aware the best method is to use an HTML parser instead but I'm really forced to apply this way

Much appreciation ,

Bass

You have participated in the culture that led to [this](http://stackoverflow.com/a/1732454/1394393). — jpmc26, Apr 05 '16 at 02:27

riteshtch · Accepted Answer · 2016-04-05T02:31:01.357

You can use something like this to hardcode match the url having municipalite and get the text inside wrt to > and < characters.

This is my data file:

 <tbody>
                      <tr>
                        <td style="width: 14%;">88055</td>
                        <td style="width: 19%;" class="gris">Ville</td>
                        <td style="width: 33%;"><a href="repertoire-des-municipalites/fiche/municipalite/88055/" >Amos</a></td>
                        <td style="width: 34%;"><a href="repertoire-des-municipalites/fiche/mrc/880/" >Abitibi</a></td>
                      </tr>
                      <tr>
                        <td style="width: 14%;">85080</td>
                        <td style="width: 19%;" class="gris">Village</td>
                        <td style="width: 33%;"><a href="repertoire-des-municipalites/fiche/municipalite/85080/" >Angliers</a></td>
                        <td style="width: 34%;"><a href="repertoire-des-municipalites/fiche/mrc/850/" >Témiscamingue</a></td>
                      </tr>
                      <tr>
                        <td style="width: 14%;">87050</td>
                        <td style="width: 19%;" class="gris">Municipalité</td>
                        <td style="width: 33%;"><a href="repertoire-des-municipalites/fiche/municipalite/87050/" >Authier</a></td>
                        <td style="width: 34%;"><a href="repertoire-des-municipalites/fiche/mrc/870/" >Abitibi-Ouest</a></td>
                      </tr>

Here is the java file:

import java.util.*;
import java.util.regex.*;
import java.lang.*;
import java.io.*;

class test
{
    public static void main (String[] args) throws java.lang.Exception
    {
        BufferedReader in  = new BufferedReader(new FileReader(new File("data")));
        String line="";
        Pattern p=Pattern.compile("href\\s*=\\s*(?:\"|').*municipalite/[^>]*>(?:<.*>)*([^<]*)<.*$");
        while ((line = in.readLine()) != null)
        {
            Matcher m=p.matcher(line);
            while(m.find())
                System.out.println(m.group(1)); 
        }
        in.close();
    }
}

Output:

$ javac test.java 
$ java test 
Amos
Angliers
Authier
$

Regular expression breakdown:

href\\s*=\\s*(?:\"|').*municipalite/[^>]*>(?:<.*>)*([^<]*)<.*$

href\\s*=\\s* matches href following by 0 or more spaces followed by = and then 0 or more spaces
(?:\"|') -> (?:) means a non capturing group i.e it matches single or double quotes but doesn't capture/remember it
.*municipalite/ matches any char till municipalite/ occurs
[^>]*>(?:<.*>)* matches any char that is not a > for the rest of the url and then matches >, then tries to match 0 or more (all optional) opening tags into a non capturing group using this (?:<.*>)
([^<]*) this group actually captures your string into group 1
<.*$ matches the rest of the line

if you use greedy matching it would match up to the last `>` skipping the in between text which we dont want — riteshtch, Apr 05 '16 at 02:25
@ritesht93 worked like a charm, thanks from the bottom of my heart — napi15, Apr 05 '16 at 02:38
@ritesh93 I meant to say non-greedy, `?` for non-greedy. Isn't it? — C Panda, Apr 05 '16 at 13:04

score 1 · Answer 2 · answered Apr 05 '16 at 02:19

1

I have shown in python. But the regex is the same in Java, I believe. Use Java functions to find the matches.

import re
reg = r"<a href=.*?municipalite.*?>(.+?)</a>"
result = re.findall(html)

answered Apr 05 '16 at 02:19

C Panda

3,297
2
11
11

score 1 · Answer 3 · answered Apr 05 '16 at 02:21

Try ".*\\bhref=\"repertoire-des-municipalites/fiche/municipalite/\\d+/\"[^>]*>([^<]*)<.*"

My demo code (below) gives console output:

Console Output

Amos
Angliers
Authier

Demo Code

public class HrefRegex
{
    public static void main(final String[] args)
    {
        final String[] sampleLines = new String[] {
            "  </tr>",
            "    <td style=\"width: 14%;\">88055</td>",
            "    <td style=\"width: 19%;\" class=\"gris\">Ville</td>",
            "    <td style=\"width: 33%;\"><a href=\"repertoire-des-municipalites/fiche/municipalite/88055/\" >Amos</a></td>",
            "    <td style=\"width: 34%;\"><a href=\"repertoire-des-municipalites/fiche/mrc/880/\" >Abitibi</a></td>",
            "  </tr>",
            "  <tr>",
            "    <td style=\"width: 14%;\">85080</td>",
            "    <td style=\"width: 19%;\" class=\"gris\">Village</td>",
            "    <td style=\"width: 33%;\"><a href=\"repertoire-des-municipalites/fiche/municipalite/85080/\" >Angliers</a></td>",
            "    <td style=\"width: 34%;\"><a href=\"repertoire-des-municipalites/fiche/mrc/850/\" >Témiscamingue</a></td>",
            "  </tr>",
            "  <tr>",
            "    <td style=\"width: 14%;\">87050</td>",
            "    <td style=\"width: 19%;\" class=\"gris\">Municipalité</td>",
            "    <td style=\"width: 33%;\"><a href=\"repertoire-des-municipalites/fiche/municipalite/87050/\" >Authier</a></td>",
            "    <td style=\"width: 34%;\"><a href=\"repertoire-des-municipalites/fiche/mrc/870/\" >Abitibi-Ouest</a></td>",
            "  </tr>",
          };


        final Pattern pattern = Pattern.compile(".*\\bhref=\"repertoire-des-municipalites/fiche/municipalite/\\d+/\"[^>]*>([^<]*)<.*");

        for (final String s : sampleLines)
        {
            final Matcher matcher = pattern.matcher(s);

            if (matcher.matches())
            {
                System.out.println(matcher.group(1));
            }
        }
    }
}

How to Extract data from HTML page using regex pattern in java

3 Answers3