how to store 25 thousand objects in Android Collection (HashMap, LinkedHashMap, ArrayList, Map)

Question

Actually I am getting 25 thousand of JSON objects from a web service and i am using Jackson to parse them and when i am trying to have that in collection(HashMap, LinkedHashMap, Array List, Map) and unfortunately using all of them i am running into "Out Of memory Exception in Android".. Can someone experimented this before with huge objects for my case it may go upto 50 thousand objects at maximum. so how do i handle in a better way. I have tried to create some customized objects by extending the above collections but i am not quite well enough to get that out. Please any suggestions.

If you have too many objects in memory then you have too many objects in memory - as long as increasing heap size is not an option. Do you receive those 25k objects from just one web service call? If so, you should consider reducing the response size (chunk) or you have to find a way to persist the objects while retrieving them... IMHO, a client device should not even handle that many objects. — home, Jan 22 '13 at 16:22
Do you have control over WebServices code? Can you page those result? You won't need them all *at once*, right? Like, you'll always show only 10 in a list or something like that? — Fildor, Jan 22 '13 at 16:29

score 4 · Answer 1 · answered Jan 22 '13 at 16:35

If you have enough memory 50 thousand or 50 million objects is not a problem. If you have limited memory, one option is to store the values on disk e.g. a memory card. You can have one file per key and the contents of the file are the value. If the values are small (< 1 KB) you might combine the values in some manner to reduce usage. As the data is text, you could compress the data to reduce disk usage as well.

You could keep a cache of recent values. This cache could be the last 10 - 1000 key/values so they can be accessed quickly with the rest on disk.

If you after compressing the data, it won't fit on the storage of the device, it is likely you will need to move the storage to a server and extract the information you need over a network.

Maybe it's a good idea to consider using the SQLite DB that Android gives us? I'm not sure that will improve the mem usage significantly. Never tried with that many objects. — Fildor, Jan 22 '13 at 16:40
If you use an SQL DB, the data stored as JSon might be redesigned to use a more SQL relational model. i.e. use it as a database instead of just a cache for a Map. — Peter Lawrey, Jan 22 '13 at 16:42

score 1 · Answer 2 · answered Jan 22 '13 at 16:37

1

Do you really need the objects in memory? You could parse the items and store them in a sqlite database. As soon as yo want to display the items fetch only the items needed from the database.

answered Jan 22 '13 at 16:37

userM1433372

5,345
35
38

score 1 · Answer 3 · answered Jan 22 '13 at 17:16

1

In Java, there are some 3party library, which has implemented; storing collection to hard disk. Like MegaMap, but on android; I do not think it's much tested.

http://megamap.sourceforge.net/index.html

As you can get 50 thousand Object, I recommend you to implement a sqlite database to store information. So that that you can fetch only needed information, whenever needed. Hope this helps.

answered Jan 22 '13 at 17:16

Munish Katoch

517
3
6

Thanks. I did divided into chuck of data – SK Ashok Dec 09 '13 at 13:18

score 0 · Accepted Answer · answered Jan 22 '13 at 16:28

0

One technique for reducing the memory foot print:

I'll talk about strings, but the following holds for other objects too.

To prevent holding the same (.equals) object in multiple instances, do something like

private Map<String, String> sharedStrings = new HashMap<>().

public void shareString(String s) {
    String t = sharedStrings.get(s);
    if (t == null) {
        t = s;
        sharedStrings.put(t, t);
    }
    return t;
}

answered Jan 22 '13 at 16:28

Joop Eggen

107,315
7
83
138

1

I guess with 25-50k Objects he runs into OutOfMem even with only one copy of the same object :) – Fildor Jan 22 '13 at 16:30
There are better String caches. Apart from that a `Set` would be more suitable than a `Map` here and the inserted Strings should be inseted with `new String(t)` to reduce memory usage when `t` was constructed by substringing of a larger string. (They share internal char[] then which is bigger than necessary.) – Fabian Barney Jan 22 '13 at 16:41
@FabianBarney I would love to see a Set for sharing; could you show how to retrieve the original string? The part on substring is certainly note worthy. – Joop Eggen Jan 22 '13 at 17:15

how to store 25 thousand objects in Android Collection (HashMap, LinkedHashMap, ArrayList, Map)

4 Answers4

Linked