Jsoup POST: Defining a selected option to return HTML?

Question

I'm trying to generate a HTML post to a page which has a list of options. There is a drop down list where the user selects an item and then clicks a generate button. Depending on the option selected it returns various results. I did a comparison of the HTML before and after and can see the differences as follows:

Before

<option value="Option1">Option 1</option>
<option value="Option2">Option 2</option>
<option value="Option3">Option 3</option>

After

<option value="Option1">Option 1</option>
<option selected="selected" value="Option2">Option 2</option>
<option value="Option3">Option 3</option>

I can't figure out the syntax for the jsoup document generation. So far I have this but it just keeps returning the original HTML with no result:

doc = Jsoup.connect("MYurl...")
.timeout(5000)
.userAgent("Mozilla/5.0 (Windows; U; WindowsNT 5.1; en-US; rv1.8.1.6) Gecko/20070725 Firefox/2.0.0.6")
.cookie("auth", "token")
.data("selected", "Option2")
.post();

score 6 · Accepted Answer · edited Jun 20 '20 at 09:12

You have to use data to set a field's value by its name attribute.

When you use:

.data("selected", "Option2")

Is the same as having this element on the requesting page form (notice the name="selected"):

<select name="selected">
    <option value="Option1">Option 1</option>
    <option selected="selected" value="Option2">Option 2</option>
    <option value="Option3">Option 3</option>
</select>

So, the way you are trying to POST is correct. The thing is probably the parameter (field) named selected does not exist in the form (that you are trying to emulate) and thus sending it is the same as sending nothing.

Edit after your comment:

What I really need to do perform a click on the web page element before parsing and from what I understand this can't be done with Jsoup.

You are correct. Jsoup doesn't support that - a click can have several side-effects like JavaScript events and so on. Processing them is a big deal.

To your specific case, though, jsoup's capabilities of altering the DOM can be useful. Check the example below. In it we "select" an option by explicitly setting its selected attribute as selected (and remove it from every other option).

import org.jsoup.Jsoup;
import org.jsoup.nodes.*;
import org.jsoup.select.*;

public class JSoupChangeDom {
    public static void main(String[] args) {
        Document doc = Jsoup.parse(""+
        " <html><body>                                            " +
        " <div>example</div>                                      " +
        " <form>                                                  " +
        "    <select name='mySelect'>                             " +
        "       <option value='Option1'>Option 1</option>         " +
        "       <option value='Option2'>Option 2</option>         " +
        "       <option value='Option3'>Option 3</option>         " +
        "    </select>                                            " +
        " <form>                                                  " +
        " </body></html>                                          ");
        Element mySelect = 
                      doc.getElementsByAttributeValue("name", "mySelect").get(0);
        String optionValueToBeSelected = "Option2";
        Elements options = mySelect.getElementsByTag("option");
        for (Element option : options) {
            if (option.attr("value").equals(optionValueToBeSelected)) {
                option.attr("selected", "selected");
            } else {
                option.removeAttr("selected");
            }
        }
        System.out.println(doc);
    }
}

Output:

<html>
    <head></head>
    <body>
        <div>example</div>
        <form>
            <select name="mySelect">
                <option value="Option1">Option 1</option>
                <option value="Option2" selected="selected">Option 2</option>
                <option value="Option3">Option 3</option>
            </select>
        </form>
    </body>
</html>

I understand what you mean but I think I've approached this issue all wrong. I figured out that I could set the attribute using Jsoup but when I post the modified HTML back to the server it just returns the option as selected. What I really need to do perform a click on the web page element before parsing and from what I understand this [can't be done with Jsoup.](http://stackoverflow.com/questions/14318991/perform-click-on-web-page-element-before-parsing-in-java) — k-prog, Jun 28 '13 at 10:21
Also because I'm trying to do this in Android it seems [this](http://stackoverflow.com/questions/9721544/httpunit-htmlunit-equivalent-for-android) is my best option but as is mentioned there it really is overkill for what I want to do. — k-prog, Jun 28 '13 at 10:31
You are correct. As an addition, I edited the answer. Check it out, it may be of help. — acdcjunior, Jun 28 '13 at 16:42

Jsoup POST: Defining a selected option to return HTML?

1 Answers1

Edit after your comment:

Linked