Firebase Database: Out of Memory Exception on querying huge amount of data on Android app

Question

I'm using Firebase database for my Android app. I needed a unique 5-digit code which should be valid for 3 days. To resolve this I inserted all possible 5-digit codes in Firebase database where each entry initially has status as 1. So structure looks something like this:

codes: {
  'abcde': {status:1},
  'zhghs': {status:1}
  .....
}

This codes object has close to 5 million entries. I was thinking I would set status to 2 once a code is in use.

I'm using this to get a code from Firebase:

db.getReference("codes")
                .orderByChild("status")
                .limitToFirst(1)
                .addListenerForSingleValueEvent { ..... }

I just need 1 code which has status as 1. The above code is causing OutOfMemoryException so I think it is trying to load all the data on device. Shouldn't it just load one element on device side due to limitToFirst(1) constraint?

P.S. I have tried my hands on Firebase FireStore. It doesn't allow importing huge JSON data as of now without exceeding 20K daily limits.

Edit - I've even set db.setPersistenceEnabled(false) Exact error is :

Firebase Database encountered an OutOfMemoryError. You may need to reduce the amount of data you are syncing to the client. java.lang.OutOfMemoryError: Failed to allocate a 28701016 byte allocation with 16777216 free bytes and 18MB until OOM

Please edit your question to limit your description to a single issue. Everything was very reasonable until you started listing a bunch of other questions at the end, which are not appropriate for Stack Overflow. — Doug Stevenson, Jul 03 '18 at 20:10
So, just to be certain, if you do nothing else in your app except for this query, you have the same problem? — Doug Stevenson, Jul 04 '18 at 04:08
@DougStevenson My app completely loads before I hit this query. Exact error is `Firebase Database encountered an OutOfMemoryError. You may need to reduce the amount of data you are syncing to the client. java.lang.OutOfMemoryError: Failed to allocate a 28701016 byte allocation with 16777216 free bytes and 18MB until OOM` — crysis, Jul 04 '18 at 05:02
The issue isn't that your app completely loads. (Maybe it loads too much, who knows?) The issue is discovering that the query actually takes too much memory without the context of the rest of your app. — Doug Stevenson, Jul 04 '18 at 05:05
@DougStevenson But the error says query is trying to load 28 MB of data. That's the size of JSON I imported in Firebase. One result shouldn't return that much data. — crysis, Jul 04 '18 at 05:14
Please edit your question to show the error that you're referring to. — Doug Stevenson, Jul 04 '18 at 05:18

score 2 · Accepted Answer · answered Jul 04 '18 at 13:45

2

It sounds like you might not have an index defined on your status property. In that case, the server can't do the ordering/filtering for you, and it will be done in the client. That means the client has to read all data and keep it in memory, explaining the huge memory usage.

The solution is quite simple: define an index on your status property.

{
  "rules": {
    "codes": {
      ".indexOn": "status"
    }
  }
}

Also see the documentation on indexing data.

answered Jul 04 '18 at 13:45

Frank van Puffelen

565,676
79
828
807

Worked like a charm. I actually had plan to add this rule but I thought I wouldn't need it in development since most of the time all status codes would be same. Thanks a lot. – crysis Jul 04 '18 at 15:54
2

With 5 million entries, you definitely need an index. Heck, with that number of entries, I'd still consider creating your own reverse index where you create the list of all items with status 1 from your own code, e.g. `/itemsByStatus/1/itemId1: true`. This scales better, since then you can order/limit the items, to get a subset: `ref.child("itemsByStatus/1").limitToFirst(10)`. – Frank van Puffelen Jul 04 '18 at 16:01

Alex Mamo · Answer 2 · 2018-07-04T18:30:52.520

1

I just need 1 code which has status as 1.

In this case change your query to:

db.getReference("codes")
            .orderByChild("status")
            .equalsTo(1)
            .limitToFirst(1)
            .addListenerForSingleValueEvent { ..... }

Even with this constraint you should reduce the amount of data you load into memory because it seems that you are forcing too much data into memory. Alternatively you can set a large heap size by setting android:largeHeap="true" in the Manifest.xml file. Usually this error disappear.

You can also take a look at the discussion in this closely related article here.

As @DougStevenson said, is not really a recommended way to solve OOM errors but as I saw in @FrankvanPuffelen answer, the key to solve the problem is to add ".indexOn": "status" which was initially missing.

edited Jul 04 '18 at 18:30

answered Jul 04 '18 at 08:02

Alex Mamo

130,605
17
163
193

Initially all codes have status = 1 so adding equalsTo(1) won't decrease the data set. Using `android:largeHeap="true"` is not the right option for me as I will shift to 6 digit code eventually where data would be much more and I don't want to load hundreds of MB data in RAM for one code. – crysis Jul 04 '18 at 10:40
Use of largeHeap is not really a recommended way to solve OOM errors, as there is still a tremendous amount of memory variance between devices, and Android is by no means obliged to give you more memory than it chooses. The correct solution is to just use less memory. https://developer.android.com/guide/topics/manifest/application-element#largeHeap – Doug Stevenson Jul 04 '18 at 17:40

Firebase Database: Out of Memory Exception on querying huge amount of data on Android app

2 Answers2

Linked