1

I am creating an app that involves reading in data from a file. The file is relatively large (1.8 MB) and is being read from an async thread in onCreate. The very first time the app is started up, it loads just fine. However, if you click the back button and then load it again, it runs out of memory and crashes (throwing the OutOfMemory error).

How do I get it to use the lowest amount of memory possible and/or free that memory when it is done?

File Reading Code (executed in the doInBackground() method of the async class):

public ArrayList<String> createDataList() {
    dataList = new ArrayList<String>();
    BufferedReader br = null;
    try {
        br = new BufferedReader(new InputStreamReader(getAssets().open(
                    "text.txt")));
        String data;
        while ((data = br.readLine()) != null) {
            dataList.add(data);
        }                           
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        try {
            br.close(); // stop reading
        } catch (IOException ex) {
            ex.printStackTrace();
        }
    }

    return dataList;
}

EDIT* Async Class:

private class loadData extends AsyncTask<Void, Void, ArrayList<String>> {

    @Override
    protected ArrayList<String> doInBackground(Void... arg0) {
        dataList = createDataList();
        return dataList;
    }

    protected void onPostExecute(ArrayList<String> result) {
        super.onPostExecute(result);
        // display first element in ArrayList in TextView as a test

    }

}

I have tried splitting up the file based on how I want to organize the data and store the data from each text file into a separate ArrayList but I had memory problems with that as well. I have also stored all of the data into one "master" ArrayList and then invoked a method on that "master" to add the data to the appropriate ArrayList (removing/clearing the data from the "master" as soon as it copied).

Any ideas on how to streamline and reduce memory impact?

EDIT**

Logcat:

enter image description here

That is from when you click the back button and then load the activity again. The following is just one of the messages produced (in verbose):

enter image description here

lord_sneed
  • 794
  • 3
  • 12
  • 25
  • What do you need the data for? What do you do with the dataList after it is returned by the method? – Mike Tunnicliffe Feb 08 '13 at 00:33
  • Are you sure the DataList is the culprit? Could you add the logcat so we can look for other messages that might show what's going on? – DigCamara Feb 08 '13 at 00:39
  • @fd. I will be displaying data to the users. Through interaction, the user will be able to display more (see my edit). – lord_sneed Feb 08 '13 at 00:39
  • @DigCamara I am pretty sure considering that fact that all I have at this point is a `TextView` to display the first element of the `ArrayList` as a test to make sure the `ArrayList` is getting populated. :) I will put up some logcat though (give me a minute to put it here). – lord_sneed Feb 08 '13 at 00:43
  • @DigCamara I added the logcat messages. – lord_sneed Feb 08 '13 at 01:02

3 Answers3

1

You can try adding android:largeHeap="true" in your manifest but it is not supported in Android API-8. To my understanding, you are reading and storing the data onto heap memory, which is usually quite limited and its size depends on the device your running your app on.

You might also want to investigate here: android - out of memory

Community
  • 1
  • 1
Barney
  • 2,355
  • 3
  • 22
  • 37
  • I do not fully understand all of the code in that link, but from what I got from the answer that was accepted the data is being written to a file. That seems counterproductive if you are reading in from a file and then writing out to another file. Or am I wrong? To get the data from the file you wrote to, you would have to read it again. So it seems like you are doing more than twice the work. Again, I have never had memory issues before because this is the first app I am making that involves reading data from a file. – lord_sneed Feb 08 '13 at 01:14
  • I could have a long method with thousands of `dataList.add();` statements to avoid reading from a file. However, that does not seem very practical, nor do I want to do that if I can avoid it. – lord_sneed Feb 08 '13 at 01:16
  • `android:largeHeap` is designed to give more heap memory to tablet applications that require extensive UI on large screens. In the Google I/O presentation about memory usage they explicitly frown upon using this feature to shortcut around writing code that is memory conscious. – Steven Byle Feb 11 '13 at 15:34
0

First, make sure you don't have two copies of the data in memory. You can null out the reference to the old ArrayList before starting to create the new one, though you have to do that carefully -- calling ArrayList.clear() first would be more thorough.

Second, figure out how much memory that ArrayList is eating up by using a tool like hprof or Eclipse MAT. If it's using a ton of space, you may want to consider a more compact data representation.

So... from the code snippet, it looks like you're just reading a bunch of text strings in from a file, using a byte-to-char conversion. If the source material is plain UTF-8 (i.e. essentially ASCII), you've got a 2x expansion to UTF-16, plus allocation of the char[] object to hold it, plus the size of the String object that wraps that, plus the overhead of the entry in the ArrayList. Depending on how long an average string in the file is, this can be a significant multiple on your 1.8MB.

One way to avoid this would be to read the file into memory as byte[], scan it to figure out where each string starts, and just keep an array of integers with the start offset of each string. When you need string N, decode it from the byte[] into a String and return it. This reduces your overhead significantly. You could reduce it further by not loading the file and just reading individual strings out as needed (using a RandomAccessFile), but this may slow things down.

fadden
  • 51,356
  • 5
  • 116
  • 166
  • I do clear the ArrayList when it is done copying the data to the right ArrayLists. The text file is in UTF-8 and the strings are no longer than 15. I am not sure how the byte array would work (I have never read a file that way before). Could you elaborate a little bit on how that works and maybe (if you could) provide an equivalent to what I have? I will think about how I could get the RandomAccessFile working (if I could for my purposes) depending on how that works. I will have to read on that too as I have not worked with that either. :) – lord_sneed Feb 08 '13 at 02:44
  • RandomAccessFile actually seems pretty decent. I honestly had never heard of it before you mentioned it. I would just have to be able to check/iterate through the strings in the file until it found the ones that met the conditions; it looks like that is possible. – lord_sneed Feb 08 '13 at 02:57
  • Can you provide an example of the RandomAccessFile? Also, where should the files go and how do you read from them? (I read that you cannot use RandomAccessFile for the assets folder.) – lord_sneed Feb 08 '13 at 04:14
  • You can find your app's private data directory from `Context.getFilesDir()` (your Activity is a Context). If you copy the data from the Asset stream to a file there, you can then use `RandomAccessFile` on the file. The trade-off is that instead of using 1.8MB of memory to hold the byte[] you're now using 1.8MB of internal disk storage, and accesses to individual strings will be slower. – fadden Feb 08 '13 at 18:22
  • I am on Linux and my main text processing program I use is gedit. I can save it as Western or UTF-8 with that. The strings range from 4-15 characters. Could provide some code as to how to copy the code to internal disk storage and then access it from RAF? I really have no idea how to do that. I am in college teaching myself this in my free time. I am not really concerned about the speed of accessing it from internal storage. I am sure it will be fast enough for my purposes (unless it takes something ridiculous like 5 seconds to get one string). – lord_sneed Feb 09 '13 at 01:26
  • Also I do not want fixed length strings because that is one of the ways by which I am organizing the data. – lord_sneed Feb 09 '13 at 01:52
0

Seems to be you might have a bit of trouble with the immutability of Strings .

Why don't you try changing your code so you use StringBuilder for instance? Of course, you'll have to change more than one thing but it would be similar enough to your code and wouldn't fill your memory up as fast.

DigCamara
  • 5,540
  • 4
  • 36
  • 47
  • How much space would be saved? I am always trying to keep everything as light and as easy on the resources as possible (for example, my backgrounds always have dark colors to save battery). Would this also fix the problem if the user hits the back button and then restarts it again? – lord_sneed Feb 08 '13 at 03:09
  • The problem is: any new String you create hangs around on the heap until... well, until all references are killed. The documentation states: "Multiple strings can share the same char[] because strings are immutable. The substring(int) method always returns a string that shares the backing array of its source string. Generally this is an optimization: fewer character arrays need to be allocated, and less copying is necessary. But this can also lead to unwanted heap retention." So, by definition you're wasting memory space by using Strings. – DigCamara Feb 08 '13 at 03:12
  • I cannot think of another way around using strings, so it seems like the RandomAccessFile might have a good chance because I would only be reading in the line(s) that I need at a given time. I could also go back to having my file be split up into separate files according to how I want them organized and I could just pass the filename as a parameter of the method the RandomAccessFile is in. I could, therefore, eliminate the need for a lot of iteration through the file. – lord_sneed Feb 08 '13 at 03:17
  • StringBuilder has a constructor that uses a String and generally is similar enough to a String so that your changes would be relatively minor. – DigCamara Feb 08 '13 at 03:20