How to get the national geographic photo of the day with jsoup in android

Question

We need to take the photo from national geographic photo of the day for an android project. We are using jsoup to do it and it is working for other sites and photos we tried to get, but not with this one. http://www.nationalgeographic.com/photography/photo-of-the-day

This is the link to the photo that we need to get. If you inspect the page, you will see that the element which contains the link, has multiple links of the photo in different sizes. So we suspect that this is the problem. Here is the element with all the links:

<source srcset="http://yourshot.nationalgeographic.com/u/fQYSUbVfts-T7odkrFJckdiFeHvab0GWOfzhj7tYdC0uglagsDq-TNIRQ3qELJppd8ZLNRvnhakVub3VQlC2V5_yAGtyNoIAtaUObf5sBn_PGVEIlVVcerfj6l1ovYy2W4h7lMAkEVLdiCZKr9S9wuwge1myLnbvmEvxjeQ-HOfdmgprhGjqn4pNtAwmKvwU6FOW3O0jR-t4LlattRw52wBmvg/ 240w, http://yourshot.nationalgeographic.com/u/fQYSUbVfts-T7odkrFJckdiFeHvab0GWOfzhj7tYdC0uglagsDq-TNIRQ3qELJppd8ZLNRvnhakVub3VQlC2V5_yAGtyNoIAtaUObf5sBn_PGVEIlVVcerfj6l1ovYy2W4h7lMAkEVLdiCZKr9S9wuwge1myLnbvmEvxjeQ-HOfdmgprhGjqn4pNtAwmKvwU6cKxp_v-TRYywK8kMonNsWFMiA/ 320w, http://yourshot.nationalgeographic.com/u/fQYSUbVfts-T7odkrFJckdiFeHvab0GWOfzhj7tYdC0uglagsDq-TNIRQ3qELJppd8ZLNRvnhakVub3VQlC2V5_yAGtyNoIAtaUObf5sBn_PGVEIlVVcerfj6l1ovYy2W4h7lMAkEVLdiCZKr9S9wuwge1myLnbvmEvxjeQ-HOfdmgprhGjqn4pNtAwmKvwU76IwFM89MgsU2CsVpABa94yrwg/ 500w, http://yourshot.nationalgeographic.com/u/fQYSUbVfts-T7odkrFJckdiFeHvab0GWOfzhj7tYdC0uglagsDq-TNIRQ3qELJppd8ZLNRvnhakVub3VQlC2V5_yAGtyNoIAtaUObf5sBn_PGVEIlVVcerfj6l1ovYy2W4h7lMAkEVLdiCZKr9S9wuwge1myLnbvmEvxjeQ-HOfdmgprhGjqn4pNtAwmKvwU7Lx-mjq8_Dk9iI7H4kcoPo-SmA/ 640w, http://yourshot.nationalgeographic.com/u/fQYSUbVfts-T7odkrFJckdiFeHvab0GWOfzhj7tYdC0uglagsDq-TNIRQ3qELJppd8ZLNRvnhakVub3VQlC2V5_yAGtyNoIAtaUObf5sBn_PGVEIlVVcerfj6l1ovYy2W4h7lMAkEVLdiCZKr9S9wuwge1myLnbvmEvxjeQ-HOfdmgprhGjqn4pNtAwmKvwU4kJMUl3WmTvlAFqfo4wIlDssvw/ 800w, http://yourshot.nationalgeographic.com/u/fQYSUbVfts-T7odkrFJckdiFeHvab0GWOfzhj7tYdC0uglagsDq-TNIRQ3qELJppd8ZLNRvnhakVub3VQlC2V5_yAGtyNoIAtaUObf5sBn_PGVEIlVVcerfj6l1ovYy2W4h7lMAkEVLdiCZKr9S9wuwge1myLnbvmEvxjeQ-HOfdmgprhGjqn4pNtAwmKvwU6-HA9n31rVvmbG5touqPt59wY3s/ 1024w, http://yourshot.nationalgeographic.com/u/fQYSUbVfts-T7odkrFJckdiFeHvab0GWOfzhj7tYdC0uglagsDq-TNIRQ3qELJppd8ZLNRvnhakVub3VQlC2V5_yAGtyNoIAtaUObf5sBn_PGVEIlVVcerfj6l1ovYy2W4h7lMAkEVLdiCZKr9S9wuwge1myLnbvmEvxjeQ-HOfdmgprhGjqn4pNtAwmKvwU6-dIS7lLTB0CSOM4O0wlvLx9pDnb/ 1600w, http://yourshot.nationalgeographic.com/u/fQYSUbVfts-T7odkrFJckdiFeHvab0GWOfzhj7tYdC0uglagsDq-TNIRQ3qELJppd8ZLNRvnhakVub3VQlC2V5_yAGtyNoIAtaUObf5sBn_PGVEIlVVcerfj6l1ovYy2W4h7lMAkEVLdiCZKr9S9wuwge1myLnbvmEvxjeQ-HOfdmgprhGjqn4pNtAwmKvwU6FcgiBNz-Nj7_J7e61F6_8oUXwoV/ 2048w" sizes="730px" data-reactid=".5.0.1.0.0.$http=2//www=1nationalgeographic=1com/photography/photo-of-the-day/2017/01/boy-buffalo-thailand.0.0.0.0.0.0.0.0">

As you can see, there are multiple links, so we also tried to split the code and get just one of them, but jsoup doesn't seem to get any of the code in the first place. Here is the code:

 Document doc = Jsoup.connect("http://www.nationalgeographic.com/photography/photo-of-the-day").get();
            Elements img = doc.select("div.modules-images__placeholder source[srcset]");

            imgSrc = img.attr("srcset"); //srcset

            String[] splitStr = imgSrc.split("\\s+");
            int n = splitStr.length;
            imgSrc = splitStr[n-2];

            //Download Image from URL
            InputStream input = new java.net.URL(imgSrc).openStream();
            //Decode Bitmap
            bitmap = BitmapFactory.decodeStream(input);
            myWallpaperManager.setBitmap(bitmap);

Diogo Correia · Answer 1 · 2017-01-21T21:23:06.760

0

I've got the solution. Also, if you want to get the different sizes, just change the 0 to another index and it will work:

String imgSrc = img.attr("srcset").split(",")[0].replaceAll(" \\d+w", "");

This is returning http://yourshot.nationalgeographic.com/u/fQYSUbVfts-T7odkrFJckdiFeHvab0GWOfzhj7tYdC0uglagsDq-TNIRQ3qELJppd8ZLNRvnhakVub3VQlC2V5_yAGtyNoIAtaUObf5sBn_PGVEIlVVcerfj6l1ovYy2W4h7lMAkEVLdiCZKr9S9wuwge1myLnbvmEvxjeQ-HOfdmgprhGjqn4pNtAwmKvwU6FOW3O0jR-t4LlattRw52wBmvg/ at the moment.

EDIT: It works if you can get the srcset attribute correctly. For some reason, Jsoup isn't getting it.

edited Jan 21 '17 at 21:23

answered Jan 21 '17 at 19:03

Diogo Correia

43
3
6

Your answer looked good, but it still doesn't seem to work. If we try running this code http://prnt.sc/dyl8z7, to get another random photo, it succesfully takes it and puts it as a background on the phone. But if we run the code with the national geographic link and your solution http://prnt.sc/dyl81r, it does nothing... – Donev Jan 21 '17 at 20:02
From what I can see you posted the same photo twice. On my answer I was assuming that img.attr("srcset") returns what's insite the srcset attribute (which I'm almost sure it does). I've not tested with the Jsoup, but it should work the same. I'll try to see what's going on once I get home. – Diogo Correia Jan 21 '17 at 20:14
Yes, I edited the comment and there are two different photos now. I suspect that the problem is that it doesn't return what's inside the srcset attribute. I am not really experienced and do not know how to check this, so if you could look into it when you are home, it will be very hopefull. Thanks. – Donev Jan 21 '17 at 20:18
@Donev to check what's the string use System.out.println(string); to print it to the debug console (if you're debugging) – Diogo Correia Jan 21 '17 at 20:23
yes, I tried that but it doesn't print anything in the debug console, even if I try to print just a random string. – Donev Jan 21 '17 at 20:40
I've just found out that they provide a JSON API to do it. Would you like me to give an answer about that or just stick with Jsoup? I've just tested with Jsoup locally and looks like it doesn't get half of the page... – Diogo Correia Jan 21 '17 at 20:54

score 0 · Answer 2 · edited May 23 '17 at 12:16

My other answer didn't work because Jsoup wasn't getting the entire page. I've found the JSON URL of the article, so I've made a simple code to get the URL:

    try {
        JSONObject jobject = readJsonFromUrl(
                "http://www.nationalgeographic.com/photography/photo-of-the-day/_jcr_content/.gallery.json");
        JSONObject article = jobject.getJSONArray("items").getJSONObject(0);
        String url = article.getString("url") + article.getString("originalUrl");
        System.out.println(url);
    } catch (Exception e) {
        e.printStackTrace();
    }

You'll need to add this methods to any class:

private static String readAll(Reader rd) throws IOException {
    StringBuilder sb = new StringBuilder();
    int cp;
    while ((cp = rd.read()) != -1) {
        sb.append((char) cp);
    }
    return sb.toString();
}

public static JSONObject readJsonFromUrl(String url) throws IOException, JSONException {
    InputStream is = new URL(url).openStream();
    try {
        BufferedReader rd = new BufferedReader(new InputStreamReader(is, Charset.forName("UTF-8")));
        String jsonText = readAll(rd);
        JSONObject json = new JSONObject(jsonText);
        return json;
    } finally {
        is.close();
    }
}

Just apply the URL to your current code and it should work.

The methods readAll and readJsonFromUrl are from this answer.

EDIT:

To get another size, use:

String url = article.getString("url") + article.getJSONObject("sizes").getString("2048");

2048 can be replaced with either 240, 320, 500, 640, 800, 1024 and 1600.

Not sure if the sizes change each day, but if so, check the JSON to see which ones are available.

where exactly do we add these functions in our android studio project? — Donev, Jan 21 '17 at 22:08
It works now, thanks a lot! Just one last question. This loads the first link in the srcset attribute, which is the small size image. How would we load some of the bigger images? — Donev, Jan 21 '17 at 22:26
Thanks for your help again! Sorry to bother you again, but I was wondering if you can tell me how to make a background process or something, that would run in the background and get the new photo every day automatically, to set it as a wallpaper, with the code we have. — Donev, Jan 22 '17 at 13:27
@Donev Take a look at [this](http://stackoverflow.com/questions/14376470/scheduling-recurring-task-in-android) answer to learn how to schedule tasks. If you want to run it async (without freezing the phone while the connection is made), take a look at the [AsyncTask](https://developer.android.com/reference/android/os/AsyncTask.html). — Diogo Correia, Jan 22 '17 at 13:58

How to get the national geographic photo of the day with jsoup in android

2 Answers2