How to use Jsoup Document method

Question

I am new and beginner in Java world. I have this code

public class Test2 {

    public static void main(String[] args) throws IOException {

    try {

        String url          = "http://www.metalbulletin.com/Login.html?ReturnURL=%2fdefault.aspx&";
            String articleURL   = "https://www.metalbulletin.com/Article/3838710/Home/CHINA-REBAR-Domestic-prices-recover-after-trading-pick-up.html";

            Connection.Response loginForm = Jsoup.connect(url)
            .method(Connection.Method.GET)
            .execute();

            Document welcomePage    = loginForm.parse();                
            Element formElement     = welcomePage.body().getElementsByTag("form").get(0);
            String formAction       = formElement.attr("action");

            Elements input = welcomePage.select("input[name=idsrv.xsrf]");
            String securityTokenValue =input.attr("value");         

            Connection.Response mainPage = Jsoup.connect("https://account.metalbulletin.com"+formAction)
            .data("idsrv.xsrf", securityTokenValue)
            .data("username", "ifiih@rupayamail.com")
            .data("password", "Kh457544")
            .cookies(loginForm.cookies())
            .method(Connection.Method.POST)
            .execute();

            Map<String, String> cookies = mainPage.cookies();

            System.out.println("\n\nloginForm.cookies()==>\n"+loginForm.cookies());
            System.out.println("\n\nmainPage.cookies()==>\n"+mainPage.cookies());

                Document articlePage    = Jsoup.connect(articleURL).cookies(cookies).get();
                Element article         = articlePage.getElementById("article-body");   
                Elements lead1          = article.getElementsByClass("articleContainer");       
                System.out.println("\n\nNews Article==>\n"+lead1);
    } catch (IOException e) {
        e.printStackTrace();
    }

    }

}

How can I refactor to this:

private Map<String, String> cookies = new HashMap<String, String>();

            private Document get(String url) throws IOException {
                Connection connection = Jsoup.connect(url);
                for (Map.Entry<String, String> cookie : cookies.entrySet()) {
                connection.cookie(cookie.getKey(), cookie.getValue());
                }
                Response response = connection.execute();
                cookies.putAll(response.cookies());
                return response.parse();
            }

I am not sure as to how I can call this private Document get(String url) method. It may seems to be stupid question but very important for me.

How can I call it within same class?

Is anybody here to help me? – Java_Beginner Nov 06 '18 at 10:27 — Java_Beginner, Nov 06 '18 at 10:27

score 0 · Answer 1 · answered Nov 14 '18 at 08:56

to do that, the easiest and more efficient solution to retrieve the Document and also the Map of Cookies, would be to create a new class called TestThreadHandler as it follows:

public class TestThreadHandler implements Runnable {

    private String url;
    private Document doc;
    private Map<String, String> cookies;
    private Semaphore barrier;

    public TestThreadHandler (String url, Document doc, Map<String, String> cookies, Semaphore barrier) {
        this.url = url;
        this.doc = doc;
        this.cookies = cookies;
        this.barrier = barrier;
    }

    public void run () {
        try {
            Connection connection = Jsoup.connect(this.url);

            for (Map.Entry<String, String> cookie : this.cookies.entrySet()) {
                connection.cookie(cookie.getKey(), cookie.getValue());
            }
            Response response = connection.execute();

            this.cookies.putAll(response.cookies());

            this.doc = response.parse();

        } catch (IOException e) {
            e.printStackTrace();
        }

        this.barrier.release();
    }

}

And call that Thread from your Test2 class from wherever you want to call it, but a sample call to that Thread would be:

public class Test2 {

    public static void main(String[] args) throws IOException {

        try {

            ...

            String url = "https://www.google.com";
            Document doc;
            Map<String, String> cookies = new HashMap<String, String>();
            Semaphore barrier = new Semaphore(0);

            Thread taskThread = new Thread( new TestThreadHandler(url, doc, cookies, barrier) );
            taskThread.start();

            barrier.acquireUninterruptibly(1); // Wait until Thread ends

            // NOW YOU HAVE BOTH DOC AND COOKIES FILLED AS DESCRIBED IN TestThreadHandler

            ...

        } catch (IOException e) {
            e.printStackTrace();
        }

    }

}

Doing that you can overwrite the variables you are passing as argument to the Thread and get both the Cookies and the JSOUP Document.

For further explanation check the Java doc of ThreadHandling or feel free to ask me!

Hope this helped you! +1

I am grateful to you. Thanks. Can you kindly update your answer and add my required code to the main method. I am very beginner to the Java — Java_Beginner, Nov 14 '18 at 11:55
@Java_Beginner you just need to implement the piece of code I wrote for the main class every time you want to retrieve a Document and the Cookies, I don't know what you want to do, so it's your task to decide when to use that method! Sorry for not being able to help you there! We all have been beginners so keep on practicing! — alvarobartt, Nov 14 '18 at 12:46
I am trying to retrieve data from metalbulletin website with simple Jsoup. Please help me to fix it — Java_Beginner, Nov 15 '18 at 11:46

How to use Jsoup Document method

1 Answers1