Log in into Xing with httpclient

Question

I've got some problems when logging in into xing with httpclient. It works when I log in manually capture the steps with livehttpheaders and reset the cookie values with addHeader("Cookie", "...."). But as soon as I logout in the browser it doesn't work anymore. I assume that I have to save the cookies and send them in every post or get method. I tried it with the BasicCookieStore but not all cookies were saved. When I run this:

BasicCookieStore cookieStore = new BasicCookieStore();
HttpContext httpContext = new BasicHttpContext();
httpContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
CloseableHttpClient httpclient = HttpClients.custom()
            .setDefaultCookieStore(cookieStore)
            .build();
try {
     HttpGet httpget = new HttpGet("https://www.xing.com/");
     CloseableHttpResponse response = httpclient.execute(httpget, httpContext);
     try {
         HttpEntity entity = response.getEntity();

         System.out.println("Login form get: " + response.getStatusLine());
         EntityUtils.consume(entity);

         System.out.println("Initial set of cookies:");
         List<Cookie> cookies = cookieStore.getCookies();
         if (cookies.isEmpty()) {
             System.out.println("None");
         } else {
             for (int i = 0; i < cookies.size(); i++) {
                 System.out.println("- " + cookies.get(i).toString());
             }
         }
     } finally {
         response.close();
     }
     HttpPost httpPost = new HttpPost("https://login.xing.com/login");
     List <NameValuePair> nvps = new ArrayList <NameValuePair>();
     nvps.add(new BasicNameValuePair("username", "name%40host.com"));
     nvps.add(new BasicNameValuePair("password", "mypassword"));
     httpPost.setEntity(new UrlEncodedFormEntity(nvps));
     httpPost.addHeader("Referer", "https://www.xing.com/");
     httpPost.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0");

     CloseableHttpResponse response1 = httpclient.execute(httpPost, httpContext);
     try {
         HttpEntity entity = response1.getEntity();
         System.out.println("Login form get: " + response1.getStatusLine());
         EntityUtils.consume(entity);

         System.out.println("Post logon cookies:");
         List<Cookie> cookies = cookieStore.getCookies();
         if (cookies.isEmpty()) {
             System.out.println("None");
         } else {
             for (int i = 0; i < cookies.size(); i++) {
                 System.out.println("- " + cookies.get(i).toString());
             }
         }
     } finally {
         response1.close();
     }

I get:

Login form get: HTTP/1.1 200 OK
Initial set of cookies:
- [version: 0][name: c_][value: 0x11791BDA9E4611E49EF65B88668E3E29][domain: .xing.com][path: /][expiry: Fri Jan 12 14:40:50 CET 2035]
Login form get: HTTP/1.1 302 Found
Post logon cookies:
- [version: 0][name: _session_id][value: 335a78c6dca029dc47e1a7d6f86e2736][domain: login.xing.com][path: /][expiry: null]
- [version: 0][name: c_][value: 0x11791BDA9E4611E49EF65B88668E3E29][domain: .xing.com][path: /][expiry: Fri Jan 12 14:40:50 CET 2035]

But I think some cookies are missing. When I log in manually there are much more cookies s_cc, s_sq, s_vi, s_fid and three which differ every time I log in: login, xing, and reg_ref Now my question: What I have to do to get these cookies and send them correctly in every method?

score 0 · Answer 1 · answered Jan 17 '15 at 14:25

You need to set the cookiepolicy for the the httpclient to process cookies, also the reason for cookie store being empty may be violation of the given cookie policy, so you have to find the best suited cookie policy.

Apache HttpClient 3.+ example

httpclient.getParams().setParameter(ClientPNames.COOKIE_POLICY, CookiePolicy.BROWSER_COMPATIBILITY);

Apache HttpClient 4.3+ example

  RequestConfig globalConfig = RequestConfig.custom().setCookieSpec(CookieSpecs.BROWSER_COMPATIBILITY).build();
CookieStore cookieStore = new BasicCookieStore();
HttpClientContext context = HttpClientContext.create();
context.setCookieStore(cookieStore);

CloseableHttpClient httpClient = HttpClients.custom().setDefaultRequestConfig(globalConfig).setDefaultCookieStore(cookieStore).build()

More about Apache HttpClient cookiepolicy here

Thank you for your answer but changing the cookiepolicy is not working for me. With `Browser_Compatibility` and `Best_Match` I receive the same cookies as posted above. With `Netscape` and `Standard` none of the cookies was found. How can i test `RFC_2109` because it does not appear in my list? Could I have another problems in my code? — Michael, Jan 17 '15 at 15:01

Log in into Xing with httpclient

1 Answers1