Json response from URL too big

Question

Trying to consume json response from url runs out of memory, or have to declare android:largeHeap="true".

This is the url

https://www.mtgjson.com/json/AllCards.json

When I enter this url in web browser on my PC, the page is populated within a second. why does it take android such long time to finish this api request? EDIT: I realize the web browser does not load all the data immediately, i see it showing the data as it comes before scrolling down in the window.

Are there ways to make the android device respond as fast as the web browser on pc when accessing this url?

When using emulators, outOfMemoryExeption unless if I increase RAM size. on samsung S7 device, outOfMemoryException unless I have largeHeap="true" in the manifest.

Same url in web browser doesn't take as long.

@Override
protected String doInBackground(String... params) {

    HttpURLConnection urlConnection = null;
    BufferedReader reader = null;
    String apiJsonStr = null;

    try {
        String baseUrl = "https://www.mtgjson.com/json/AllCards.json";

        String uri = Uri.parse(baseUrl).buildUpon().toString();

        URL url = new URL(baseUrl);
        urlConnection = (HttpURLConnection) url.openConnection();
        urlConnection.setRequestMethod("GET");
        urlConnection.connect();
        InputStream inputStream = urlConnection.getInputStream();
        StringBuffer buffer = new StringBuffer();
        if (inputStream == null) {
            return null;
        }
        reader = new BufferedReader(new InputStreamReader(inputStream));
        String line;
        while ((line = reader.readLine()) != null) {
            buffer.append(line + "\n");
        }

        if (buffer.length() == 0) {
            return null;
        }
        apiJsonStr = buffer.toString();
    } catch (IOException e) {
        return null;
    } finally {
        if (urlConnection != null) {
            urlConnection.disconnect();
        }
        if (reader != null) {
        }
    }
    return apiJsonStr;
}

@Override
protected void onPostExecute(String stringReturn) {
    super.onPostExecute(stringReturn);

    mListener.allCardsResponse(stringReturn);

}

Looks like a duplicate of https://stackoverflow.com/questions/25803443/how-to-read-big-json-conten-from-url — DCTID, Sep 05 '19 at 23:30
Yes, that is the answer, it worked with StringBuilder! I tried searching and did not find that answer. thank you — DevJZ, Sep 05 '19 at 23:58
The StringBuilder worked, but still only with good internet connection and the device still needed a large heap. I ended up creating a intermediary back end like one of the suggestions in the answer below. — DevJZ, Sep 08 '19 at 19:39

score 1 · Answer 1 · answered Sep 06 '19 at 00:05

Before you venture that route of trying to fit a square peg in a round hole by using a big hammer to smash the hole. I suggest you think of a better approach.

First of all, I looked at the json document. My browser took 1 min to load it and browser these days have gotten pretty efficient so they start to stream chunks they receive immediately. Second the file in 65.5 MB uncompressed.

This is a disaster ready to happen when you publish the app. User experience is going to suck. Everyone internet speed is going to be different and not to mention their connection proximity to your server (network latency). If it's going to take forever to load this thing if it's happening synchronously which it looks like it is, user are going to abandon the app.

Here is what I propose you look into -

If you have control over URL which returns the JSON, see if you can use pagination and only return a fraction that you are going to display in your mobile app. I'm pretty sure you can't cram all these information a 6' - 9' mobile screen.
Perhaps you can transform this endpoint to only return few interesting metadata and when users show interest, you pull down all the details of that specific metadata (which will be one json object in your list)? That's assuming you control the API.
If this is an API that you don't control, you can create an intermediary service that will do #1 for you so you can better optimize it.
Another approach if you don't control the URL and the content is static enough you can package with your app or configure out a background sync that chunks these into 1 MB each and works asynchronously.

Thank you. I ended up doing the intermediary service. I have Firebase functions downloading the file, and taking the one string from each item and caching that. Now my application hits firebase for a much much smaller json response. Firebase will only have to re-download the file every time new sets are released about every 3 months. Thank you. I did have the original post in an Async task however. The new small response is very fast. I wanted the list of every card on the device db with room, so AutoCompleteTextView can efficiently search the db as the user types. — DevJZ, Sep 08 '19 at 19:37
@JZudell Glad to hear that you opted for a reasonable solution to the problem. Cheers! — na-98, Sep 09 '19 at 19:34

Json response from URL too big

1 Answers1