Problems using Jsoup to retrieve sessionId cookies

Question

I'm to write a Java program, which in part, parses 200 unique pages that require the user to log in beforehand. I've used Chrome's developer console to determine my specific login URL(https://r.espn.go.com/members/v3_1/login), verify that the login process used a POST request, and the Form Data names for both my username (username) and password (password). When using the method specified by the author of this post to retrieve a SESSIONID cookie for subsequent requests the returned headers were vastly different and no cookies were returned.

I've also tried the following snippet which uses both Jsoup and Apache's HttpClient, HttpPost, and HttpResponse which returns the loginpage:

MultipartEntity entity = new MultipartEntity();
entity.addPart("username", new StringBody(myUsername));
entity.addPart("password", new StringBody(myPassword));

HttpPost post = new HttpPost(url);
post.setEntity(entity);

HttpClient client = new DefaultHttpClient();
HttpResponse response = client.execute(post);

String html = EntityUtils.toString(response.getEntity());

Document document = Jsoup.parse(html, url);

Every example I've read have a login url with a .php suffix, does this method only work with a PHP based login service? Or am I doing something fundamentally wrong?

Thanks!

Answer 1

Let HttpClient manage the cookies/session for you. For this to happen

1. Create a HttpContext and use it for every request you make so that the session/cookie management is active.
2. Set up the cookie store
3. Exucute each web request within the context you created in step 1

Below is the sample code for HttpClient 4.1.x version. Read Section 3.8 HTTP state management and execution context of their documentation. Also, go through this thread.

//create the local context to be shared across multiple requests of the same session
HttpContext localContext = new BasicHttpContext(); 

 // Create a local instance of cookie store
CookieStore cookieStore = new BasicCookieStore();

// Bind custom cookie store to the local context
localContext.setAttribute(HttpClientContext.COOKIE_STORE, cookieStore);

// execute the post within the context
HttpResponse response = client.execute(post,localContext);

If this didn't solve the issue then use wireshark or Fiddler2 to inspect the HTTP request and response traffic.