Android OutOfMemoryException when processing large JSON response

Question

I am getting an OutOfMemoryException when processing a very long JSON response returned by the code below:

HttpClient httpclient = new DefaultHttpClient();
HttpGet httpGETRequest = new HttpGet(url);
HttpResponse response = httpclient.execute(httpGETRequest);
return EntityUtils.toString(response.getEntity());

Even though I use System.gc() before I call the http request, the exception still occurs.

I tried to look for the cause of the problem by saving the response on a notepad and check its size, just to find out that it is a 5mb file.

What are the things that should be done in order to avoid the exception?

Thanks for your help in advance.

@Haspemulator, the last line gives the OOM.. Specifically the EntityUtils.toString(); — Rick Royd Aban, May 13 '15 at 15:00
Okay, then it doesn't matter if it's JSON or not, because you're not doing the JSON parsing here, just loading the relevant String into memory. Plus, `HttpClient` is deprecated, use `URLConnection` instead. — Haspemulator, May 13 '15 at 15:04

score 2 · Accepted Answer · answered Mar 02 '17 at 00:29

2

In this cause, we should avoid large data responses from server. I requested a pagination functionality from the server and problem solved.

answered Mar 02 '17 at 00:29

Rick Royd Aban

904
6
33

Archimedes Trajano · Answer 2 · 2015-05-13T15:09:01.823

I don't recommend that you use System.gc() on Android. What you would need to do is use streaming and avoid loading the whole content in memory if it can be avoided.

https://google-gson.googlecode.com/svn/trunk/gson/docs/javadocs/com/google/gson/JsonStreamParser.html

If you need to access parts of the JSON data more readily rather than doing it one chunk at at time, you can transform the data as it is being read and store it in a SQL database that you can query later.

http://developer.android.com/training/basics/data-storage/databases.html

However, if you have embedded binary data such as an image as part of the JSON object you can't do anything about it anymore because it will read the embedded binary data into memory as one chunk. There's no easy way around this aside from writing your own JSON streamer.

Generally

This general technique can be applied not just to Android, but even application servers dealing with large input data such as XML as well.

score 1 · Answer 3 · edited May 23 '17 at 12:09

1

Calling System.GC() is not guaranteed to run GC. In fact its not a good practice to programmatically invoke GC. For parsing large json data, use streaming APIs in whichever library you use to encode/decode json. One such popular library is Jackson processor.

5mb is not that big data to cause out of memory, maybe dump the memory of the app and analyze for memory leaks. You could sump using adb dumpsys Tool. Find more about how to use that in What's the Android ADB shell "dumpsys" tool and what are its benefits?

edited May 23 '17 at 12:09

Community

1
1

answered May 13 '15 at 15:02

Thilak

656
7
15

Yes I am aware that System.gc() is not guaranteed to run the garbage collection but I did that to improve the chances of avoiding the OOM. I am quite confused about the difference of streaming from reading JSON response.... – Rick Royd Aban May 13 '15 at 15:08
When using json Parsers, streaming APIs provide low memory footprint as such APIs don't create additional intermediate Java objects, which copies the actual reaponse – Thilak May 13 '15 at 15:28

score 0 · Answer 4 · answered May 13 '15 at 15:07

0

try {
        HttpResponse response = httpclient.execute(httppost);
        HttpEntity entity = response.getEntity();
        inputStream = entity.getContent();
        // json is UTF-8 by default
        BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"), 8);
        StringBuilder sb = new StringBuilder();
        String line = null;
        while ((line = reader.readLine()) != null) {
            sb.append(line + "\n");
        }
        result = sb.toString();
        inputStream.close();
    } catch (Exception e) {
        Log.d(TAG, "String was to large" + e.toString());
    }

answered May 13 '15 at 15:07

Xjasz

1,238
1
9
21

Thanks for you answer, I tried this and same exception occurs in 'while ((line = reader.readLine()) != null) {' – Rick Royd Aban May 13 '15 at 15:09
You may need to parse it your own way then. You shouldn't be running into a memory issue with a String though. I would make a parser that breaks up the String into smaller objects then Rick. gl – Xjasz May 13 '15 at 15:14
How big is this String really. If its that big you should be parsing it as it reads line by line and storing it into a sqlLite db or an object. – Xjasz May 13 '15 at 15:17
This is just a band-aid solution, that doesn't solve the problem. – Martin Konecny May 13 '15 at 20:18

score 0 · Answer 5 · answered May 13 '15 at 15:12

I think if you have problem to load 5Mb string into memory (totally reasonable size), streaming JSON parser will probably not help, because you need to have the result of parsing in memory, which will not necessarily be much smaller. I'd suggest you to do the memory profiling using Android Studio built-in memory monitor, to check how much free memory you have and how you're using it in general.

Android OutOfMemoryException when processing large JSON response

5 Answers5

Generally