Need help to understand memory leak in my Android app

Question

My app runs fine until I interrupt the initialization process at the very first start after installation by exiting and launching the app several times as long as the initialization process has not yet been finished. The processing logic and the AsyncTask can handle this pretty well, so I don't get any inconsistencies, but I have a problem with the heap. It increasing more and more while I do this disturbing exits and launches at app setup, which will lead to OutOfMemory error. I already found a leak by analyzing the heap with MAT but I still have another leak which I can't isolate yet.
For background info: I store the application context, a list and a timestamp in a static class to be able to access it from classes anywhere in my application without using tedious passing references by constructor. Anyway, there must be something wrong with this static class (ApplicationContext) since it causes a memory leak due to the list of zones. Zone objects are processed GeoJSON data. This is how this class looks like:

public class ApplicationContext extends Application {
    private static Context context;
    private static String timestamp;
    private static List<Zone> zones = new ArrayList<Zone>();

    public void onCreate()  {
        super.onCreate();
        ApplicationContext.context = getApplicationContext();
    }

    public static Context getAppContext() {
        return ApplicationContext.context;
    }

    public static List<Zone> getZones() {
        return zones;
    }

    public static void setData(String timestamp, List<Zone> zones) {
        ApplicationContext.timestamp = timestamp;
        ApplicationContext.zones = zones;
    }

    public static String getTimestamp() {
        return timestamp;
    }
}

I already tried to store the zones like this

ApplicationContext.zones = new ArrayList(zones);

but it had no effect. I already tried to put the zones attribute into another static class since ApplicationContext is loaded before all other classes (due to an entry in AndroidManifest) which could lead to such behavior but this isn't the problem too.

setData is invoked in my "ProcessController" twice. Once in doUpdateFromStorage, and once in doUpdateFromUrl(String). This class looks like this:

public final class ProcessController {
    private HttpClient httpClient = new HttpClient();

    public final InitializationResult initializeData()  {
        String urlTimestamp;
        try {
            urlTimestamp = getTimestampDataFromUrl();

            if (isModelEmpty())  {
                if (storageFilesExist())  {
                    try {
                        String localTimestamp = getLocalTimestamp();

                        if (isStorageDataUpToDate(localTimestamp, urlTimestamp))  {
                            return doDataUpdateFromStorage();
                        } 
                        else  {
                            return doDataUpdateFromUrl(urlTimestamp);
                        }
                    } 
                    catch (IOException e) {
                        return new InitializationResult(false, Errors.cannotReadTimestampFile());
                    }
                }
                else  {
                    try {
                        createNewFiles();

                        return doDataUpdateFromUrl(urlTimestamp);
                    } 
                    catch (IOException e) {
                        return new InitializationResult(false, Errors.fileCreationFailed());
                    }
                }
            }
            else  {
                if (isApplicationContextDataUpToDate(urlTimestamp))  {
                    return new InitializationResult(true, "");  
                }
                else  {
                    return doDataUpdateFromUrl(urlTimestamp);
                }
            }
        } 
        catch (IOException e1) {
            return new InitializationResult(false, Errors.noTimestampConnection());
        }
    }

    private String getTimestampDataFromUrl() throws IOException {
        if (ProcessNotification.isCancelled()) {
            throw new InterruptedIOException();
        }

        return httpClient.getDataFromUrl(FileType.TIMESTAMP);
    }

    private String getJsonDataFromUrl() throws IOException {
        if (ProcessNotification.isCancelled()) {
            throw new InterruptedIOException();
        }

        return httpClient.getDataFromUrl(FileType.JSONDATA);
    }

    private String getLocalTimestamp() throws IOException {
        if (ProcessNotification.isCancelled()) {
            throw new InterruptedIOException();
        }

        return PersistenceManager.getFileData(FileType.TIMESTAMP);
    }

    private List<Zone> getLocalJsonData() throws IOException, ParseException {
        if (ProcessNotification.isCancelled()) {
            throw new InterruptedIOException();
        }

        return JsonStringParser.parse(PersistenceManager.getFileData(FileType.JSONDATA));
    }

    private InitializationResult doDataUpdateFromStorage() throws InterruptedIOException {
        if (ProcessNotification.isCancelled()) {
            throw new InterruptedIOException();
        }

        try {
            ApplicationContext.setData(getLocalTimestamp(), getLocalJsonData());

            return new InitializationResult(true, "");
        } 
        catch (IOException e) {
            return new InitializationResult(false, Errors.cannotReadJsonFile());
        } 
        catch (ParseException e) {
            return new InitializationResult(false, Errors.parseError());
        }
    }

    private InitializationResult doDataUpdateFromUrl(String urlTimestamp) throws InterruptedIOException {
        if (ProcessNotification.isCancelled()) {
            throw new InterruptedIOException();
        }

        String jsonData;
        List<Zone> zones;
        try {
            jsonData = getJsonDataFromUrl();
            zones = JsonStringParser.parse(jsonData);

            try {
                PersistenceManager.persist(jsonData, FileType.JSONDATA);
                PersistenceManager.persist(urlTimestamp, FileType.TIMESTAMP);

                ApplicationContext.setData(urlTimestamp, zones);

                return new InitializationResult(true, "");
            } 
            catch (IOException e) {
                return new InitializationResult(false, Errors.filePersistError());
            }
        } 
        catch (IOException e) {
            return new InitializationResult(false, Errors.noJsonConnection());
        } 
        catch (ParseException e) {
            return new InitializationResult(false, Errors.parseError());
        }
    }

    private boolean isModelEmpty()  {
        if (ApplicationContext.getZones() == null || ApplicationContext.getZones().isEmpty())  {    
            return true;
        }

        return false;
    }

    private boolean isApplicationContextDataUpToDate(String urlTimestamp) { 
        if (ApplicationContext.getTimestamp() == null)  {
            return false;
        }

        String localTimestamp = ApplicationContext.getTimestamp();

        if (!localTimestamp.equals(urlTimestamp))  {
            return false;
        }

        return true;
    }

    private boolean isStorageDataUpToDate(String localTimestamp, String urlTimestamp) { 
        if (localTimestamp.equals(urlTimestamp))  {
            return true;
        }

        return false;
    }

    private boolean storageFilesExist()  {
        return PersistenceManager.filesExist();
    }

    private void createNewFiles() throws IOException {
        PersistenceManager.createNewFiles();
    }
}

Maybe it's another helpful information, that this ProcessController is invoked by my MainActivity's AsyncTask at the app setup:

public class InitializationTask extends AsyncTask<Void, Void, InitializationResult> {
    private ProcessController processController = new ProcessController();
    private ProgressDialog progressDialog;
    private MainActivity mainActivity;
    private final String TAG = this.getClass().getSimpleName();

    public InitializationTask(MainActivity mainActivity) {
        this.mainActivity = mainActivity;
    }

    @Override
    protected void onPreExecute() {
        super.onPreExecute();

        ProcessNotification.setCancelled(false);

        progressDialog = new ProgressDialog(mainActivity);
        progressDialog.setMessage("Processing.\nPlease wait...");
        progressDialog.setIndeterminate(true); //means that the "loading amount" is not measured.
        progressDialog.setCancelable(true);
        progressDialog.show();
    };

    @Override
    protected InitializationResult doInBackground(Void... params) {
        return processController.initializeData();
    }

    @Override
    protected void onPostExecute(InitializationResult result) {
        super.onPostExecute(result);

        progressDialog.dismiss();

        if (result.isValid())  {
            mainActivity.finalizeSetup();
        }
        else  {
            AlertDialog.Builder dialog = new AlertDialog.Builder(mainActivity);
            dialog.setTitle("Error on initialization");
            dialog.setMessage(result.getReason());
            dialog.setPositiveButton("Ok",
                    new DialogInterface.OnClickListener() {

                        @Override
                        public void onClick(DialogInterface dialog, int which) {
                            dialog.cancel();

                            mainActivity.finish();
                        }
                    });

            dialog.show();
        }

        processController = null;
    }

    @Override
    protected void onCancelled() {
        super.onCancelled();

        Log.i(TAG, "onCancelled executed");
        Log.i(TAG, "set CancelNotification status to cancelled.");

        ProcessNotification.setCancelled(true);

        progressDialog.dismiss();

        try {
            Log.i(TAG, "clearing files");

            PersistenceManager.clearFiles();

            Log.i(TAG, "files cleared");
        } 
        catch (IOException e) {
            Log.e(TAG, "not able to clear files.");
        }

        processController = null;

        mainActivity.finish();
    }
}

Here is the body of the JSONParser. (UPDATE: I set the method none static but the problem persists.) I omit the object creations from the JSON objects since I don't think that this is the error:

public class JsonStringParser {
    private static String TAG = JsonStringParser.class.getSimpleName();

    public static synchronized List<Zone> parse(String jsonString) throws ParseException, InterruptedIOException {
        JSONParser jsonParser = new JSONParser();

        Log.i(TAG, "start parsing JSON String with length " + ((jsonString != null) ? jsonString.length() : "null"));
          List<Zone> zones = new ArrayList<Zone>();

        //does a lot of JSON parsing here

        Log.i(TAG, "finished parsing JSON String");

        jsonParser = null;

        return zones;
    }
}

Here is the heap dump which shows the problem:

This is the details list which shows that this problem has something to do with the arraylist.

Any ideas what's wrong here? Btw: I don't know what's the other leak since there is no details information.

Maybe important: This diagram show the status when I don't start and stop the application over and over again. It's a diagram of a clean start. But when I start and stop several times it could lead to problems due to lack of space.

Here is a diagram of a real crash. I started and stopped the app while initialization several times:

[UPDATE]
I narrowed it down a bit by not storing the Android context into my ApplicationContext class and making PersistenceManager non-static. The problem hasn't changed, so I'm absolutely sure that it is not related to the fact that I store the Android context globally. It's still "Problem Suspect 1" of the graph above. So I have to do something with this huge list, but what? I already tried to serialize it, but unseralizing this list takes much longer than 20secs, so this is not an option.

Now I tried something different. I kicked out the whole ApplicationContext so I don't have any static references anymore. I tried to hold the ArrayList of Zone objects in MainActivity. Although I refactored at least the parts I need to make the application run, so I didn't even pass the Array or the Activity to all classes where I need it, I still have the same problem in a different manner, so my guess is that the Zone objects itself are somehow the problem. Or I cannot read the heap dump properly. See the new graphs below. This is the result of a simple app start without interference.

[UPDATE]
I came to the conclusion that there is no memory leak, because "the memory is accumulated in one instance" doesn't sound like a leak. The problem is that starting and stopping over and over again starts new AsyncTasks, as seen on one graph, so the solution would be to not start new AsyncTask. I found a possible solution on SO but it doesn't work for me yet.

Answer 1

First of all, I have to agree with Emile:

The "..tedious passing references by constructor" is what helps avoid issues like this. Honestly, using statics in this way is certainly one way to create memory leaks like this, especially with a static reference to your context.

This also applies to all those other static methods in your code. static methods are not really different from global functions. You are building a big spaghetti plate full of static methods there. Especially when they start sharing some state it will sooner or later crash or create obscure results which you wouldn't get with a proper design, especially in the presence of a highly multi-threadable platform as Android.

What also catched my eye is, please note that the onCancelled method of the AsyncTask will not be called before doInBackground has finished. So your global cancelation flag (ProcessNotification.isCancelled()) is more or less worthless (if only used in the shown code passages).

Also from the memory images you posted, the zones list contains "only" 31 items. How much is it supposed to hold? By how much does it increase? If it actually increases, the culprint might be in the JsonStringParser.parse method, which is again static. If it holds a list of items in some cache and the control logic is not working correctly (for example in the presence of multiple threads accessing it at the same time), it might add items to that cache each time it is called.

• Guess 1: As the parsing method is static, this data is not (necessarily) cleaned when the application is shut down. statics are initialized once and for the purpose of this case never de-initialized until the (physical vm-)process is stopped. Android does not guarantee that the process is killed however, even if the application is stopped (see for example a wonderful explanation here). Therefore you might accumulate some data in some static part of your (maybe parsing) code.
• Guess 2: Since you are re-starting your application several times, you have the background thread running several times in parallel (assumption: each time you restart the application a new thread is spawned. Note that your code shows no guards against this.) This the first parsing is still running, another one is started since the global zones variables still holds no values. The global function parse might not be thread-safe and put several data multiple times into the list which is eventually returned, yielding a bigger and bigger list. Again this is generally avoided by not having static methods (and be aware of multi-threading).

(The code is not complete, therefore guesses, there might even be other things lurking there.)

Answer 2

Inside your AsyncTask, you own a reference on a Context : MainActivity. When you start several AsyncTask, they are gonna be queued by an ExecutorService. So all the AsyncTask, if they are long running, will be "alive" (not garbage collected). And each of them will keep a reference on an Activity. Consequently, all you activities are gonna be kept alive as well.

This is a real memory leak as Android will want to garbage collect an Activity that is not displayed any more. And your AsyncTasks will prevent that. All the activities are kept in memory.

I encourage you to try RoboSpice Motivations to learn more about this problem. In this app we explain why you should not use AsyncTasks for long running operations. There are still a few work around that enable you to use them, but they are difficult to implement.

One way to get rid of this problem is to use WeakReference to point to your Activities inside your AsyncTask class. If you use them carefully, you can then avoid your activities not to be garbage collected.

Actually, RoboSpice is a library that allows to execute Network requests inside a service. This approach is quite interesting has it will create a context (a service) that is not linked to your activities. Thus, your request can take as long as they want and don't interfere with the garbage collection behavior of Android.

There are two modules of RoboSpice that you can use to deal with REST request. One for Spring Android and the other one for Google Http Java Client. Both libs will ease JSON parsing.

Answer 3

I assume you fixed the reference to MainActivity, but I'd like to mention another problem ...

You state that the parsing takes 20sec. And if you "interrupt" the app, this processing does not go away.

From the code you show here it seems 99% of that 20sec is spent inside JsonStringParser.parse().

If I look at your comment "does a lot of JSON parsing here", I assume your app makes a call into JSONParser.something() that stays away for 20sec. Even though JsonStringParser is static, each call to JsonStringParser.parse() creates a new copy of JSONParser() and my guess is that uses a lot of memory.

A background process that takes 20sec is a really big task, and in what I have seen with JSON parsers, in this time a lot of objects get created and destroyed and a lot of cycles get consumed.

So I think your root cause here is that you start a second (or third or fourth) copy of JSONParser.something(), because each of them will execute independently and try to allocate many chunks of memory, and stay running even longer than 20sec because they will have to share the CPU cycles. The combined memory allocation of multiple JSONParser objects is what kills your system.

To summarize:

• Do not start another JsonStringParser.parse() until the first one is killed or completed.
• This means you must find a way to stop JsonStringParser.parse() when you "interrupt" the app, or reuse the running copy when you restart the app.

Answer 4

THink i see how it might be possible, my eyes have gone crossed eyed looking though.

Check that your not loading the data from your local storage, adding more data to it and then saving it back to local disk.

Something around the following methods in combination with other parts of your program.

If the following was called, and then you call getDatafromURL for some reason, then i believe you'd continually grow your data set.

That would be my starting point at least. Loading, appending and saving.

ApplicationContext.setData(getLocalTimestamp(), getLocalJsonData());

private List<Zone> getLocalJsonData() throws IOException, ParseException {
    if (ProcessNotification.isCancelled()) {
        throw new InterruptedIOException();
    }

    return JsonStringParser.parse(PersistenceManager.getFileData(FileType.JSONDATA));
}

Otherwise i think the problem lies in either your Parsing code, or perhaps one of the static classes your using to save the data.

Answer 5

MY FINAL SOLUTION

I found a solution on my own now. It runs stable and doesn't produce memory leaks when I start and stop the application a lot of times. Another advantage with this solution is that I was able to kick out all this ProcessNotification.isCancelled() parts.

The key is to hold a reference to my InitializationTask in my ApplicationContext. With this approach I can resume the running AsyncTask in a new MainActivity when I start a new one. This means that I never start more than one AsyncTask but I attach every new MainActivity instance to the currently running task. The old Activity will be detached. This looks like this:

new methods in ApplicationContext:

public static void register(InitializationTask initializationTask) {
    ApplicationContext.initializationTask = initializationTask;
}

public static void unregisterInitializationTask()  { 
    initializationTask = null;
}

public static InitializationTask getInitializationTask() {
    return initializationTask;
}

MainActivity
(I have to put the progressDialog in here, otherwise it wouldn't be shown if I stop and start a new Activity):

@Override
protected void onStart() {
    super.onStart();

    progressDialog = new ProgressDialog(this);
    progressDialog.setMessage("Processing.\nPlease wait...");
    progressDialog.setIndeterminate(true); // means that the "loading amount" is not measured.
    progressDialog.setCancelable(true);
    progressDialog.show();

    if (ApplicationContext.getInitializationTask() == null) {
        initializationTask = new InitializationTask();
        initializationTask.attach(this);

        ApplicationContext.register(initializationTask);

        initializationTask.execute((Void[]) null);
    } 
    else {
        initializationTask = ApplicationContext.getInitializationTask();

        initializationTask.attach(this);
    }
}

MainActivity's "onPause" contains initializationTask.detach(); and progressDialog.dismiss();. finalizeSetup(); dismisses the dialog too.

InitializationTask contains two more methods:

public void attach(MainActivity mainActivity) {
    this.mainActivity = mainActivity;
}

public void detach() {
    mainActivity = null;
}

onPostExecute of the task invokes ApplicationContext.unregisterInitializationTask();.