Login using Jsoup

Question

I would like to login into a website and keep the cookies. I have issues with the login part using Jsoup;

My code ;

                   Connection.Response loginForm = Jsoup.connect("URL" + "/login/").cookies(cookies).method(Connection.Method.GET).execute();

        formData.put("username", "#######");
        formData.put("pwd", "########");
        formData.put("hidden","69");
        formData.put("token", loginForm.parse().select("input#token").first().attr("value"));

        Connection.Response homePage = Jsoup.connect("URL" + "/login/")
                .referrer("URL" + "/login/")
                .followRedirects(true)
                .cookies(cookies)
                .data(formData)
                .method(Connection.Method.POST)
                .execute();

        cookies_2.putAll(homePage.cookies()); // save the cookies, this will be passed on to next request

If i go to the login page and use the developper tool there are ;

Google chrome developper tool

Google chrome developper tool

Edit 1;

The problem now is I get homepage of the website but without the section of login, I keep the "Connect or register" button.

I output the cookies_2 and it is the same that the cookies in the Chrome logs ;

Jsoup cookies_2 output

Chrome cookies when login

Now, what I don't understand is why, I don't get loggin if I have the right cookies ?

Edit 2;

I chanded my code with the final solution and it work !

Thanks for your help !

Answer 1

Few remarks:

• You're getting HTTP status 302. It may be a good idea to use Jsoup.connect(...).followRedirects(true)
• Some servers check the site you're coming from so it's recommened to set referrer header: Jsoup.connect(...).referrer(URL + "/login")
• Are you sure the hidden value is always 69? Maybe it's different for each request. You can get it dynamically like this: formData.put("hidden",html.select("co_js").first().attr("value"));

• I don't like your way of getting the token. Let's use Jsoup to extract it: String authToken = html.select("input#token").first().attr("value");

  Edit:

• I tried to do this from scratch and submitted login form was missing Content-Type definition. Try using: Jsoup.connect(...).header("Content-Type","application/x-www-form-urlencoded") I got the idea by analyzing headers in Chrome's dev tools. Now I can successfully login.
• You can also check if you're logged in using this code: Document doc = homePage.parse(); System.out.println("Logged in as: " + doc.select(".dropdown-toggle").text());

• By letting server know you can handle compressed pages you can decrease downloaded page size. For every request use: Jsoup.connect(...).header("accept-encoding", "gzip, deflate") It's transparent and you don't have to do anything special to handle it, but it works internally.

  Edit 2:

Providing final solution based on previous advices:

    import java.util.HashMap;
    import java.util.Map;

    import org.jsoup.Connection;
    import org.jsoup.Connection.Response;
    import org.jsoup.Jsoup;
    import org.jsoup.nodes.Document;

    public class Stackoverflow51734840 {

        private static final String URL = "https://time2watch.in";
        private static final String URL_LOGIN = URL + "/login/";
        static String userAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36";

        public static void main(final String[] args) throws Exception {

            Map<String, String> headers = new HashMap<String, String>();
            headers.put("accept-encoding", "gzip, deflate");

            Connection.Response loginForm = Jsoup.connect(URL + "/login").userAgent(userAgent).headers(headers).execute();
            Map<String, String> cookies = loginForm.cookies();
            Document html = loginForm.parse();

            String authToken = html.select("input#token").first().attr("value");
            System.out.println("Found authToken:" + authToken);

            Map<String, String> formData = new HashMap<String, String>();
            formData.put("username", "!!!!!!!!!!!!!!!!!!");
            formData.put("pwd", "!!!!!!!!!!!!!!!!!!");
            formData.put("hidden", "69");
            formData.put("token", authToken);
            headers.put("Content-Type", "application/x-www-form-urlencoded");

            System.out.println("cookies before login:");
            System.out.println(cookies);
            System.out.println(" Logged in cookie present? " + cookies.containsKey("s4icookuser"));

            Connection.Response afterLoginPage = Jsoup.connect(URL_LOGIN).cookies(cookies).headers(headers)
                    .userAgent(userAgent).data(formData).method(Connection.Method.POST).referrer(URL_LOGIN).execute();
            // update cookies
            cookies = afterLoginPage.cookies();

            System.out.println("cookies after login:");
            System.out.println(cookies);
            System.out.println(" Logged in cookie present? " + cookies.containsKey("s4icookuser"));

            Response homePage = Jsoup.connect(URL).cookies(cookies).method(Connection.Method.GET).userAgent(userAgent)
                    .referrer(URL_LOGIN).followRedirects(true).referrer(URL_LOGIN).headers(headers).execute();

            Document doc = homePage.parse();
            System.out.println("Error? " + doc.text().contains("Erreur"));
            System.out.println("OK? " + !doc.text().contains("Se connecter"));
            System.out.println("Logged in as: " + doc.select(".dropdown-toggle").text());
        }

    }