Can't access some DOM elements using Jsoup

Question

I'm trying to access this website : https://casino.betin.co.ke/virtual/Virtual+Football using Jsoup to get text from it's elements but I get errors. Here is my code snippet:

public static final String URL = "https://casino.betin.co.ke/virtual/Virtual+Football";

public void getTitle() {
    StringRequest titleRequest = new StringRequest(Request.Method.GET, URL,
            new Response.Listener<String>() {
                @Override
                public void onResponse(String response) {
                    Document document = Jsoup.parse(response);
                    Element element = document.select("div.matchlist-header").first();
                    Toast.makeText(mContent, element.text(), Toast.LENGTH_SHORT).show();
                }
            },
            new Response.ErrorListener() {
                @Override
                public void onErrorResponse(VolleyError error) {
                    error.printStackTrace();
                }
            });

This is the error I get in android studio

java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String org.jsoup.nodes.Element.text()' on a null object reference

It looks like NPE is thrown on `element.text()` which suggest that `element` holds `null`. Since it was result of `Element element = document.select("div.matchlist-header").first();` it suggests that there is no element in DOM held by `document` which matches CSS query `div.matchlist-header` which is why `first()` returns `null`. Make sure that response you got from server really contains structure you are looking for. If such HTML content is generated dynamically by JS then jsoup is not tool for the job since it doesn't support JS (it is HTML parser, not browser emulator). — Pshemo, Mar 28 '17 at 15:36
Possible duplicate of [Jsoup Java HTML parser : Executing javascript events](http://stackoverflow.com/questions/7344258/jsoup-java-html-parser-executing-javascript-events) — Pshemo, Mar 28 '17 at 15:37
What developer tools? The ones from browser? That doesn't show what originally was send to client (browser), but what it holds *currently* (including content generated/loaded by JavaScript later). Try printing `document.toString()` to see what actually is send to client and compare both structures. — Pshemo, Mar 28 '17 at 18:06
That is why jsoup can't find element which you are looking for. You need other tool, capable of executing JavaScript. Take a look at suggested duplicate. — Pshemo, Mar 28 '17 at 18:18
@shmakova said that previously. Thanks. Let me try the other tools — Oirere Jr, Mar 28 '17 at 18:21

shmakova · Accepted Answer · 2017-03-28T15:20:49.460

0

Jsoup is a html parser only. Unfortunately it's not possible to parse any javascript / ajax content, since Jsoup can't execute those.

You can disable javascript on the page and try to reload it. You will see that div.matchlist-header element doesn't exist

edited Mar 28 '17 at 15:20

answered Mar 28 '17 at 15:14

shmakova

6,076
3
28
44

Can't access some DOM elements using Jsoup

1 Answers1