JSONArray throwing out of memory exception in android

Question

In my app I sync some data at the end of day to the app server.For this I wrap all my data as a JSONArray of JSONObjects.The data mainly includes about 50 pictures each with a size of approx 50kb(along with some text data).All these pictures are encoded using base64 encoding.Everthing works fine when the pictures uploaded(along with some text data) are few in number,but when I upload a large no of pictures ,say around 50 then I see in the logs that all the data is properly formed into the JSONArray,however when I try to display the JSONArray using 'array.toString()' method I encounter an out of memory exception.This I believe is due to the heap getting full(however,when I try making android:largeHeap="true" in the manifest everything is working fine,however I want to avoid using this approach,since this is not a good practice).My intention is just to write this JSONArray value into a file and then break this file into small chunks and send it across to the server. Please guide me of the best approach of writing the JSONAray value to the file which won't lead to OOM issues.Thanks !

Following is the format of the JSONArray:

[{"pid":"000027058451111","popup_time":"2014-01-13 23:36:01","picture":"...base64encoded string......","punching_time":"Absent","status":"Absent"},{"pid":"000027058451111","popup_time":"2014-01-13 23:36:21","picture":"...base64encoded string......","punching_time":"Absent","status":"Absent"}]

Following are the main snippets of my code:

            JSONObject aux;
            JSONArray array = new JSONArray();
            .
            .
            // Looping through each record in the cursor
            for (int i = 0; i < count; i++) {
                aux = new JSONObject();

                try {
                    aux.put("pid", c.getString(c.getColumnIndex("pid")));
                    aux.put("status", c.getString(c.getColumnIndex("status")));
                    aux.put("pop_time", c.getString(c.getColumnIndex("pop_time")));
                    aux.put("punching_time", c.getString(c.getColumnIndex("punching_time")));
                    aux.put("picture", c.getString(c.getColumnIndex("image_str"))); // stores base64encoded picture
                } catch (Exception e) {
                    e.printStackTrace();
                }

                array.put(aux); // Inserting individual objects into the array , works perfectly fine,no error here
                c.moveToNext(); // Moving the cursor to the next record
            }

            Log.d("Log", "length of json array - "+array.length()); // shows me the total no of JSONObjects in the JSONArray,works fine no error

            // HAD GOT OOM HERE
            //Log.d("Log", "JSONArray is - " + array.toString()); 

            if (array.length() != 0){
                try {

                    String responseCode = writeToFile(array);  //Writing the JSONArray value to file,which will then send file to server.

                    if(responseCode.equals("200"))
                        Log.d("Log","Data sent successfully from app to app server");
                    else    
                        Log.d("Log","Data NOT sent successfully from app to app server");
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
            .
            .

            private String writeToFile(JSONArray data) {

            Log.d("Log", "Inside writeToFile");
            File externalStorageDir = new File(Environment.getExternalStorageDirectory().getPath(), "Pictures/File");

            if (!externalStorageDir.exists()) {
                externalStorageDir.mkdirs();
            }

            String responseCode = "";
            File dataFile = new File(externalStorageDir, "File");
    /*      FileWriter writer;
            String responseCode = "";
            try {
                writer = new FileWriter(dataFile);
                writer.append(data);
                writer.flush();
                writer.close();

                responseCode = sendFileToServer(dataFile.getPath(), AppConstants.url_app_server); // Sends the file to server,worked fine for few pictures

            } catch (IOException e) {
                e.printStackTrace();
            }*/


            try {
                FileWriter file = new FileWriter("storage/sdcard0/Pictures/File/File");

                file.write(data.toString());        // GOT OOM here.
                file.flush();
                file.close();
                Log.d("Log","data  written from JSONArray to file");
                responseCode = sendFileToServer(dataFile.getPath(), AppConstants.url_app_server);    // Sends the file to server,worked fine for few pictures
            } catch (IOException e) {
                e.printStackTrace();
            }

            return responseCode;

        }


        public String sendFileToServer(String filename, String targetUrl) {
            .
            .
            // Sends the file to server,worked fine for few pictures
            .
            .
            return response;
        }

Answer 1

Here's the issue. You're trying to load your entire dataset into memory. And you're running out of memory.

Android's JSON classes (and some other JSON libraries) are designed to take a Java object (in memory), serialize it to a parse tree of objects (e.g. JSONObject, JSONArray) (in memory), then convert that tree to a String (in memory) and write it out somewhere.

Specifically in your case (at the moment) it appears what when it converts the parse tree into a String it runs out of memory; That String is effectively doubling the amount of memory required at that point.

To solve your issue you have a few different choices, I'll offer 3:

• Don't use JSON at all. Refactor to simply send files and information to your server.

• Refactor things so that you only read X images into memory at a time and have multiple output files. Where X is some number of images. Note this is still problematic if your image sizes vary greatly / aren't predictable.

• Switch to using Jackson as a JSON library. It supports streaming operations where you can stream the JSON to the output file as you create each object in the array.

Edit to add: for your code, it would look something like this using Jackson:

// Before you get here, have created your `File` object
JsonFactory jsonfactory = new JsonFactory();
JsonGenerator jsonGenerator = 
    jsonfactory.createJsonGenerator(file, JsonEncoding.UTF8);

jsonGenerator.writeStartArray();

// Note: I don't know what `c` is, but if it's a cursor of some sort it
// should have a "hasNext()" or similar you should be using instead of
// this for loop
for (int i = 0; i < count; i++) {

    jsonGenerator.writeStartObject();

    jsonGenerator.writeStringField("pid", c.getString(c.getColumnIndex("pid")));
    jsonGenerator.writeStringField("status", c.getString(c.getColumnIndex("status")));
    jsonGenerator.writeStringField("pop_time", c.getString(c.getColumnIndex("pop_time")));
    jsonGenerator.writeStringField("punching_time", c.getString(c.getColumnIndex("punching_time")));
    // stores base64encoded picture
    jsonGenerator.writeStringField("picture", c.getString(c.getColumnIndex("image_str")));

    jsonGenerator.writeEndObject();

    c.moveToNext(); // Moving the cursor to the next record
}

jsonGenerator.writeEndArray();
jsonGenerator.close();

The above is untested, but it should work (or at least get you going in the right direction).

Answer 2

First and foremost.Thanks a Billion to Brian Roach for assisting me.His inputs helped me solve the problem.I am sharing my answer.

What was I trying to solve? - In my project I had some user data(name,age,picture_time) and some corresponding pictures for each of the user data.At the EOD I needed to sync all this data to the app server.However when I tried to sync a lot of pictures(say 50 of 50kb approx) I faced an OOM(Out of Memory) issue.Initially, I was trying to upload all the data using a conventional JSONArray approach,however soon I found that I was hitting OOM.This, I attribute to the heap getting full when I was trying to access the JSONArray(which had loads of values and why not ?, afterall I was encoding the pics by base64encoding,which trust me has a hell lot of string data in it !)

Inputs from Brian suggested that I write all my data into a file one by one.So,after the whole process is complete I get one single file that has all the data(name,age,picture_time,base64encoded pictures etc) in it,and then I stream this file to the server.

Following is the code snippet which takes the user data from app database,corresponding pictures from sd card,loops through all the records,creates a JSONArray of JSONObjects using Jackson Json Library(which you need to include in your libs folder,should you use this code) and stores them into a file.This file is then streamed to the server(this snippet not included).Hope this helps someone!

---

            // Sync the values in DB to the server
            Log.d("SyncData", "Opening db to read files");
            SQLiteDatabase db = context.openOrCreateDatabase("data_monitor", Context.MODE_PRIVATE, null);
            db.execSQL("CREATE TABLE IF NOT EXISTS user_data(device_id VARCHAR,name VARCHAR,age VARCHAR,picture_time VARCHAR);");
            Cursor c = db.rawQuery("SELECT * FROM user_data", null);

            int count = c.getCount();

            if (count > 0) {

                File file = new File(Environment.getExternalStorageDirectory().getPath(), "Pictures/UserFile/UserFile");
                JsonFactory jsonfactory = new JsonFactory();
                JsonGenerator jsonGenerator = null;
                try {
                    jsonGenerator = jsonfactory.createJsonGenerator(file, JsonEncoding.UTF8);
                    jsonGenerator.writeStartObject();       
                    jsonGenerator.writeArrayFieldStart("user_data"); //Name for the JSONArray
                } catch (IOException e3) {
                    e3.printStackTrace();
                }

                c.moveToFirst();

                // Looping through each record in the cursor
                for (int i = 0; i < count; i++) {               

                    try {

                        jsonGenerator.writeStartObject();  //Start of inner object '{'
                        jsonGenerator.writeStringField("device_id", c.getString(c.getColumnIndex("device_id")));
                        jsonGenerator.writeStringField("name", c.getString(c.getColumnIndex("name")));
                        jsonGenerator.writeStringField("age", c.getString(c.getColumnIndex("age")));
                        jsonGenerator.writeStringField("picture_time", c.getString(c.getColumnIndex("picture_time")));


                    } catch (Exception e) {
                        e.printStackTrace();
                    }

                    // creating a fourth column for the input of corresponding image from the sd card

                    Log.d("SyncData", "Name of image - " + c.getString(c.getColumnIndex("picture_time")));

                        image = c.getString(c.getColumnIndex("picture_time")).replaceAll("[^\\d]", ""); //Removing everything except digits
                        Log.d("SyncData", "imagename - " + image);

                        File f = new File(Environment.getExternalStorageDirectory().getPath(), "Pictures/UserPic/" + image + ".jpg");
                        Log.d("SyncData", "------------size of " + image + ".jpg" + "= " + f.length());
                        String image_str;

                        if (!f.exists() || f.length() == 0) {
                            Log.d("SyncData", "Image has either size of 0 or does not exist");
                            try {
                                jsonGenerator.writeStringField("picture", "Error Loading Image");
                            } catch (Exception e) {
                                e.printStackTrace();
                            }
                        } else {

                            try {

                                // Reusing bitmaps to avoid Out Of Memory
                                Log.d("SyncData", "Image exists,encoding underway...");
                                if (bitmap_reuse == 0) {    //ps : bitmap reuse was initialized to 0 at the start of the code,not included in this snippet

                                    // Create bitmap to be re-used, based on the size of one of the bitmaps
                                    mBitmapOptions = new BitmapFactory.Options();
                                    mBitmapOptions.inJustDecodeBounds = true;
                                    BitmapFactory.decodeFile(f.getPath(), mBitmapOptions);
                                    mCurrentBitmap = Bitmap.createBitmap(mBitmapOptions.outWidth, mBitmapOptions.outHeight, Bitmap.Config.ARGB_8888);
                                    mBitmapOptions.inJustDecodeBounds = false;
                                    mBitmapOptions.inBitmap = mCurrentBitmap;
                                    mBitmapOptions.inSampleSize = 1;
                                    BitmapFactory.decodeFile(f.getPath(), mBitmapOptions);

                                    bitmap_reuse = 1;
                                }

                                BitmapFactory.Options bitmapOptions = null;

                                // Re-use the bitmap by using BitmapOptions.inBitmap
                                bitmapOptions = mBitmapOptions;
                                bitmapOptions.inBitmap = mCurrentBitmap;

                                mCurrentBitmap = BitmapFactory.decodeFile(f.getPath(), mBitmapOptions);

                                if (mCurrentBitmap != null) {
                                    ByteArrayOutputStream stream = new ByteArrayOutputStream();
                                    try {
                                        mCurrentBitmap.compress(Bitmap.CompressFormat.JPEG, 35, stream);
                                        Log.d("SyncData", "------------size of " + "bitmap_compress" + "= " + mCurrentBitmap.getByteCount());

                                    } catch (Exception e) {
                                        e.printStackTrace();
                                    }

                                    byte[] byte_arr = stream.toByteArray();
                                    Log.d("SyncData", "------------size of " + "image_str" + "= " + byte_arr.length);

                                    stream.close();
                                    stream = null;

                                    image_str = Base64.encodeToString(byte_arr, Base64.DEFAULT);

                                    jsonGenerator.writeStringField("picture", image_str);

                                }

                            } catch (Exception e1) {
                                e1.printStackTrace();
                            }
                        }

                    try {
                        jsonGenerator.writeEndObject();  //End of inner object '}'
                    } catch (JsonGenerationException e) {
                        e.printStackTrace();
                    } catch (IOException e) {
                        e.printStackTrace();
                    }

                    c.moveToNext(); // Moving the cursor to the next record
                }

                try {
                    jsonGenerator.writeEndArray();      //close the array ']'
                    //jsonGenerator.writeStringField("file_size", "0");   // If need be, place another object here.
                    jsonGenerator.writeEndObject();     

                    jsonGenerator.flush();
                    jsonGenerator.close();

                } catch (JsonGenerationException e1) {
                    e1.printStackTrace();
                } catch (IOException e1) {
                    e1.printStackTrace();
                }

                c.close();
                db.close();
            }