0

I am facing two problems when using JSoup to scrape data from the web:

  1. It's performance is not that good: it takes a bit too long to connect to a URL.

  2. For some sites, it's not fetching the correct data from the URL. For example, try any URL of the NY Times, like — http://www.nytimes.com/2014/06/13/technology/facebook-to-let-users-alter-their-ad-profiles.html?ref=technology

It just loads the login page, but when I try the same URL on the Google or Facebook, they correctly fetch the data. Also, the URL loads fine in a browser for a guest user.

anotherdave
  • 6,656
  • 4
  • 34
  • 65
Satya Narayan Patidar
  • 41
  • 9
  • 2
    NYT requires that your application is able to deal with coockies - otherwise your request gets redirected to the login page as far as I know. Similar threads on this topic: [Extract NYT articles using JSoup](http://stackoverflow.com/questions/24078520/extract-new-york-times-articles-using-jsoup) and [How to manage cookies using JSoup](http://stackoverflow.com/questions/9069288/how-to-manage-cookies-with-jsoup). Also this [link](http://stackoverflow.com/questions/6432970/jsoup-posting-and-cookie) might be of interest for you – Roman Vottner Jun 13 '14 at 11:24

1 Answers1

1

What's happening here is that the NY Times is using a session cookie to determine if they should show you the content, or redirect you to the login page.

Because JSoup is dropping the cookies, you'll always retrieve the log-in page, rather than getting sent back to the content.

According to the JSoup docs, you can retrieve the cookies on the response using the cookies() method.

You can then set them on your next request with the cookies(Map<String, String>) method on Connection.

You can then manage the cookies in your request/response chain.

This isn't specifically a JSoup issue, you can reproduce the same thing with curl on the command line:

curl -v http://www.nytimes.com/…

Returns a "See other" request with the location of the login page

curl -v http://www.nytimes.com/glogin?URI=http%3A%2F%2Fwww.nytimes.com%2F…

Drops a cookie & gives a "302" request to send you back to the page.

If I request the page again, I'll start the process over, unless I send their session cookie along with my request.

anotherdave
  • 6,656
  • 4
  • 34
  • 65