How to extract elements from html page using HtmlUnit

Question

I have two questions(problems actually) while parsing the html page using HtmlUnit.I tried their 'Getting Started` as well as searched google but no help.Here is my first problem.

1) I want to extract the text of following bold tag from the page

<b class="productPrice">Five Dollars</b>

2)I want to extract the entire text(including further span or link text,if present) in the last paragraph in the following structure

<div class="alertContainer">
<p>Hello</p>
<p>Haven't you registeret yet?</p>
<p>Registrations will close on 3 July 2012.<span>So don't wait</span></p>
</div>

Can you please one-line code snippets how can I do that?I am new to HtmlUnit.

EDIT:

HtmlUnit has getElementByName() and getElementById(), so what do we use if we want to select using class?

This will be the answer to my first question.

Have you tried `getElementsByAttribute()` and `getOneHtmlElementByAttribute()`? (where the attributeName is "class") — MattR, Jan 22 '14 at 09:46

score 7 · Accepted Answer · edited Feb 08 '21 at 09:56

actually, I'd suggest you to use xpath and jtidy instead, like this

import java.io.IOException;
import java.net.MalformedURLException;
import java.util.List;

import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlForm;
import com.gargoylesoftware.htmlunit.html.HtmlItalic;
import com.gargoylesoftware.htmlunit.html.HtmlOption;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import com.gargoylesoftware.htmlunit.html.HtmlRadioButtonInput;
import com.gargoylesoftware.htmlunit.html.HtmlSelect;
import com.gargoylesoftware.htmlunit.html.HtmlSubmitInput;
import com.gargoylesoftware.htmlunit.html.HtmlTextArea;
import com.gargoylesoftware.htmlunit.html.HtmlTextInput;

public class WebScraper {

    private static final String TEXT = "some random text here";
    private static final String SWALLOW = "continental";
    private static final String COLOR = "indigo2";
    private static final String QUESTION = "why?";
    private static final String NAME = "Leo";

    /**
     * @param args
     * @throws IOException
     * @throws MalformedURLException
     * @throws FailingHttpStatusCodeException
     */
    public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException {
   
        //to get the HTML Xpath, download and install firefox plugin Xpather from
        //http://jassage.com/xpather-1.4.5b.xpi
        //
        //then right-click on any part of the html and choose "show in xpather"
        //
        //HtmlUnit is a suite for functional web app tests (headless) with a
        //built-in "browser". Very useful for screen scraping.
        //
        //for HtmlUnit examples and usage, try
        //http://htmlunit.sourceforge.net/gettingStarted.html
        //
        //sometimes, the HTML is malformed, so you'll need to "clean it"
        //that's why I've also added JTidy to this project
       
        WebClient webClient = new WebClient();
       
        HtmlPage page = webClient.getPage("http://cgi-lib.berkeley.edu/ex/simple-form.html");
       
//        System.out.println(page.asXml());
       
        HtmlForm form = (HtmlForm) page.getByXPath("/html/body/form").get(0);
       
        HtmlTextInput name = form.getInputByName("name");
        name.setValueAttribute(NAME);
       
        HtmlTextInput quest = form.getInputByName("quest");
        quest.setValueAttribute(QUESTION);
       
        HtmlSelect color = form.getOneHtmlElementByAttribute("select", "name", "color");
        List<HtmlOption> options = color.getOptions();
        for(HtmlOption op:options){
            if (op.getValueAttribute().equals(COLOR)){
                op.setSelected(true);
            }
        }
       
        HtmlTextArea text = form.getOneHtmlElementByAttribute("textarea", "name", "text");
        text.setText(TEXT);
       
        //swallow
        HtmlRadioButtonInput swallow = form.getInputByValue(SWALLOW);
        swallow.click();
       
        HtmlSubmitInput submit = form.getInputByValue("here");

        //submit
        HtmlPage page2 = submit.click();
       
//        System.out.println(page2.asXml());
       
        String color2 = ((HtmlItalic)page2.getByXPath("//dd[1]/i").get(0)).getTextContent();
        String name2 = ((HtmlItalic)page2.getByXPath("//dd[2]/i").get(0)).getTextContent();
        String quest2 = ((HtmlItalic)page2.getByXPath("//dd[3]/i").get(0)).getTextContent();
        String swallow2 = ((HtmlItalic)page2.getByXPath("//dd[4]/i").get(0)).getTextContent();
        String text2 = ((HtmlItalic)page2.getByXPath("//dd[5]/i").get(0)).getTextContent();

        System.out.println(COLOR.equals(color2)
                && NAME.equals(name2)
                && QUESTION.equals(quest2)
                && SWALLOW.equals(swallow2)
                && TEXT.equals(text2));
       
        webClient.closeAllWindows();

    }

}

:Can we use regex in HtmlUnit?Like the problem is `
` Now that `832` may change so can I use something like `fresh_article_[0-9]*` in my xpath? or what's the alternative. — Naveen, Jan 22 '14 at 10:12
I don't know if httpunit xpath supports fn:matches() but you could give it a try :-) — , Jan 22 '14 at 10:22
LeonardoKenji:Thanks for xpather again.And for the question I asked in comment.It worked too.Actually the trick is to leave out the id and access the element(text) using the parent container (like div,p,b,etc). Answer Accepted — Naveen, Jan 22 '14 at 11:09
:I got into a problem.The html is malformed so I am getting wrong result.You mentioned JTidy.How can I use that? — Naveen, Jan 22 '14 at 13:08
http://stackoverflow.com/questions/10390922/how-do-i-make-jtidy-make-html-documents-well-formed — , Jan 22 '14 at 13:23

How to extract elements from html page using HtmlUnit

1 Answers1