Firebase Realtime Database: Using "indexOn" with Nested Dynamic Id?

Question

I have written some delete queries for a Realtime Database in Firebase, but keep running into issues due to a lack of indexing.

Here is an example of the data structure for one of the collections:

"followers": {
  "user_id1": {
     "user_ida": {
       "field1": "string val",
       "field2": true,
       "user_id": "user_ida"
     },
     "user_idb": {
       "field1": "string val",
       "field2": false,
       "user_id": "user_idb"
     }
  }
  "user_id2": {
    "user_idc": {
       "field1": "string val",
       "field2": true,
       "user_id": "user_idc"
     },
     "user_idd": {
       "field1": "string val",
       "field2": false,
       "user_id": "user_idd"
     }
  }
}

I want to query the collection by the nested user_ids, so that I can delete users from the lists that are no longer in our system. In my staging environment, the query works, and looks like this::

const userId = 'my_user_id_input';

const rt_ref = await app.database().ref('followers')
  .orderByChild(`${userId}/user_id`)
  .equalTo(userId)

const rt_proc = new BatchProcessor(rt_ref);
const rt_proms = [];

await rt_proc.forEach(key_id => {
  console.log(`key_id = ${key_id}`);
  rt_proms.push(app.database()
          .ref(collectionName)
          .child(key_id)
          .child(userId)
          .remove()
          .then(() => println(`    Successfully deleted user_id from Realtime "followers" where parent_id was ${key_id}`))
          .catch((err) => {
            println(`    Error deleting Realtime "followers" for parent ${key_id} where ${userId} is child id: ${err}`);
          }));
});

return Promise.all(rt_proms);

However ... because part of the query is dynamic -- ${userId}/user_id -- this results in a warning to add an ".indexOn" rule to my followers collection in the Realtime database::

@firebase/database: FIREBASE WARNING: Using an unspecified index. Your data will be downloaded and filtered on the client. Consider adding ".indexOn": "my_user_id_input/user_id" at /followers to your security rules for better performance.

I've tried throwing everything at this in the Rules, but I've not been able to get anything to work. I'm stumped. Do I need to structure my query differently?

Here's what I've got at the moment in the Rules for this collection:

/* NOTE: FIGURE OUT INDEX FOR NESTED USER_ID! */
    "followers": {
      "$followed_id": {
        "$follower_id": {
          ".write": "(auth != null)",
          ".indexOn": ["user_id", ".value"]
        },
        ".indexOn": [
          "timestamp",
          "user_name",
          "user_id",
          ".value"
        ]
      },
      ".read": "true",
      ".indexOn": ["user_id", ".value"]
    },

Granted, there are way too many "indexOn" entries in that collection, but I'm just trying as much as I can to see what will work.

I need to have this index to work, because we have a large amount of data in our PRODUCTION database, and this code just times out when it is run there :(.

Is there a solution for what I am trying to accomplish?

Frank van Puffelen · Answer 1 · 2023-07-28T19:59:46.647

0

Firebase Realtime Database can only query on a flat list of nodes, where the value to order/filter on is at a fixed path under each direct child node. That means that the .orderByChild(${userId}/user_id) query in your use-case is not possible, unless you can enumerate all possible values of userId in your security rules.

What you have is an index that makes it easy for you to find the followers for a given UID. But the use-case you have is to find the followees for a given UID, which your current data structure does not allow you to do.

What you need for this is an additional so-called reverse index, pretty much the exact inverse of what you have now. For the data you have, that'd be:

"followees": {
  "user_ida": {
    "user_id1": true
  },
  "user_idb": {
    "user_id1": true
  },
  "user_idc": {
    "user_id2": true
  },
  "user_idd": {
    "user_id2": true
  },
}

And with that you can find the followees under /followees/$userId.

Also see:

Many to Many relationship in Firebase

edited Jul 28 '23 at 19:59

answered Jul 28 '23 at 01:50

Frank van Puffelen

565,676
79
828
807

Thank you for this fast and detailed response, @Frank! It's good to know there's no good way for me to get to those child ids. Just, grrr, how to remove them when it's time to purge inactive users?? – Miss Henesy Jul 28 '23 at 02:01
Just in code, by performing the inverse operation of when you create them. – Frank van Puffelen Jul 28 '23 at 02:28
I'm going to have to sit and dwell on this a bit. We do have a "following" collection, but I do not think that we connect "followers" and "following" in any useful way, so, unless I forcibly remove the child user_ids, deleted users still appear to follow active users. I will take a deeper dive into Many to Many relationships (Firebase style -- I'm well versed in "normal" dbs :-) ) -- and see if I can wrap my head around your advice. Thank you so much for helping me battle through this logic! – Miss Henesy Jul 28 '23 at 17:48
Whatever the names of the nodes: you have a structure that seem to model "who do I follow?", but a use-case "who follows me?". In NoSQL databases you often need to change/expand your data model for the use-cases of your app, hence adding the inverse data here. --- If you're new to NoSQL, read [NoSQL data modeling](https://highlyscalable.wordpress.com/2012/03/01/nosql-data-modeling-techniques/) and watch [Firebase for SQL developers](https://www.youtube.com/playlist?list=PLl-K7zZEsYLlP-k-RKFa7RyNPa9_wCH2s). – Frank van Puffelen Jul 28 '23 at 19:59

Firebase Realtime Database: Using "indexOn" with Nested Dynamic Id?

1 Answers1