Optimize firebase realtime database search

Question

Below is my table structure in firebase realtime database. I have an array of ids and want to check if those ids exists in any of these refs data/userId/tests1 or data/userId/tests2. If the id don't exist in both if these ref I am adding it to my resultFilteredUsers array. Below is the solution I currently have but this is very slow. Is there any faster approach than this?

data
    userId
        tests1
            id1
            id2
        tests2
            id3
            id4

code

async function filter(userId: String, testIdsArray:string[]) {
    
        var filteredUsersSet1: string[] = []
        var resultFilteredUsers: string[] = []
    
        var db = FirebaseAdmin.database()
        var ref1 = db.ref("data/" + userId + "/tests1/")
        var ref2 = db.ref("data/" + userId + "/tests2/")
    
        await Promise.all(
            testIdsArray.map(id => {
                return ref1.child(id).once('value')
                  .then(snapshot => {
                    if (!snapshot.exists()){
                        filteredUsersSet1.push(id)
                    }
                    return snapshot;
                  }).catch((error: any) => {
                      logger.error("Error");
                    });
                })
            )
    
        await Promise.all(
            filteredUsersSet1.map(id => {
                return ref2.child(id).once('value')
                .then(snapshot => {
                    if (!snapshot.exists()){
                        resultFilteredUsers.push(id)
                    }
                    return snapshot;
                  }).catch((error: any) => {
                      logger.error("Error");
                    });
                })
            )
        return resultFilteredUsers;
    }

score 0 · Answer 1 · answered Sep 08 '22 at 20:55

0

Since you're using await for the two Promise.all calls, you are ensuring that they happen sequentially. I don't think your use-case requires that in any way, so it'd be faster to combine all those promises in a single Promise.all call:

const promises1 = testIdsArray.map(id => {
    return ref1.child(id).once('value')
      .then(snapshot => {
        if (!snapshot.exists()){
            filteredUsersSet1.push(id)
        }
        return snapshot;
      }).catch((error: any) => {
          logger.error("Error");
        });
    })
)
    
const promises2 = filteredUsersSet1.map(id => {
    return ref2.child(id).once('value')
    .then(snapshot => {
        if (!snapshot.exists()){
            resultFilteredUsers.push(id)
        }
        return snapshot;
      }).catch((error: any) => {
          logger.error("Error");
        });
    })
)

await Promise.all(promises1.concat(promises2));

The above is more efficient, but won't be orders of magnitude faster.

answered Sep 08 '22 at 20:55

Frank van Puffelen

565,676
79
828
807

I did sequential of two promise calls to avoid firebase realtime database costs and as per my design, an id can exist only in either of those two refs and can never be in both. If you see, I am passing filteredUsersSet1 which will have the data that doesn't exist in ref1 from promise call 1 as an input to second promise call. – marsuser Sep 09 '22 at 17:39
Ah, I see now. I recommend not creating such side-effects going forward, but returning the database results from the first `Promise.all` so that the flow is more obvious. That said, you're still waiting until you've gotten all users, before you then check their existence under `ref2` - something you can actually do for each user as soon as you've loaded them. That will allow more parallelism too, although here too not by orders of magnitudes. Also see: https://stackoverflow.com/questions/35931526/speed-up-fetching-posts-for-my-social-network-app-by-using-query-instead-of-obse/35932786#35932786 – Frank van Puffelen Sep 09 '22 at 19:22
A way to speed this up much more would be to duplicate the data for each user under `ref2` into their node under `ref1` already. – Frank van Puffelen Sep 09 '22 at 19:23

Optimize firebase realtime database search

1 Answers1