Best way to query relational data in firebase

Question

I'm currently developing an admin dashboard using Firebase, where I need to fetch data for all members, including their related payments and vehicles.

Here is the data model I'm working with - overview of data schema using Typescript types

export type VehicleDocType = {
  // Vehicle document fields
  owner: string; // Member id
  ...
};

export type PaymentDocType = {
  // Payment document fields
  recipient: string; // Member id
  ...
};

export type MemberDocType = {
  uid: string;
  ...
  /* Note! Arrays are not coming from Firestore and not even part of Schema.
   it is used to map relations on the client side
  */
  payments: Array<PaymentDocType> | null;
  vehicles: Array<VehicleDocType> | null;
};

Data structure in Firebase:

----------------------------------------------------------
| Collections |   Documents | Fields                      |
----------------------------------------------------------
| members     |   n docs    | { uid: doc.id, ...}         |
| payments    |   n docs    | { recipient: member.id, ... } |
| vehicles    |   n docs    | { owner: member.id, ... }     |
----------------------------------------------------------

Note: vehicles collection is the same as payments.

For the member page, I want to display all members along with their associated payments and vehicles.

This is my current approach for fetching data:

Firebase SDK: "firebase": "^9.22.0"

Exact queries I want to run:

const fetchCustomers = async () => {
  // Fetch member documents from the 'membersCol' collection
  const memberDocs = (await getDocs(membersCol)).docs.map((d) => d.data());

  // Fetch payment documents from the 'paymentsCol' collection
  const paymentDocs = (await getDocs(paymentsCol)).docs.map((d) => d.data());

  // Fetch vehicle documents from the 'vehiclesCol' collection
  const vehicleDocs = (await getDocs(vehiclesCol)).docs.map((d) => d.data());

  
  const members = memberDocs.map((member) => ({
    ...member,
    payments: paymentDocs.filter((p) => p.recipient === member.uid),
    vehicles: vehicleDocs.filter((v) => v.owner === member.uid),
  }));

  // Further processing and rendering of the data
};

While this approach is functional, I have encountered a few problems:

• Loading large datasets can result in slow data retrieval times.
• When implementing Firebase pagination queries, I'm unsure how to efficiently map the relations (vehicles and payments) to each member.
• I'm looking for best practices to fetch the data with minimal document reads and optimize the overall loading time.

Any suggestions or recommendations on how to improve the data fetching process, optimize document reads, and reduce loading time would be greatly appreciated. Furthermore, if someone have any recommendations on how to implement real-time updates in this situation.

Answer 1

Based on your current approach, you can marginally optimize the sorting of payments and vehicles by attaching each payment and vehicle directly to the member.

For an array of 10 (M) members, 20 (V) vehicles and 30 (P) payments:

• 60 document reads will be billed each time a user loads the page and they miss the cache
• the map members, then filter payments/vehicles approach from your question will invoke the filter callbacks a total of 500 times (M * (V + P)). Additionally, the filter callbacks need to be initialised again inside each map callback increasing the processing time.
• the sorting forEach approach below will invoke the forEach callbacks only 50 times (V + P).

As you can see from the above calculation, it is M times more efficient to use the latter approach.

const fetchCustomers = async () => {
  // Fetch member documents from the 'membersCol' collection
  const memberMap = (await getDocs(membersCol)).docs
    .reduce(
      (map, d) => map.set(
        d.id, // maybe needs to be d.get("uid") if uid <> docId
        { ...d.data(), payments = [], vehicles = [] }
      ),
      new Map()
    );

  // Fetch payment documents from the 'paymentsCol' collection
  const allPayments = (await getDocs(paymentsCol)).docs.map((d) => d.data());

  // Fetch vehicle documents from the 'vehiclesCol' collection
  const allVehicles = (await getDocs(vehiclesCol)).docs.map((d) => d.data());

  allPayments.forEach(pData => {
    memberMap.get(pData.recipient).payments.push(pData);
  });

  allVehicles.forEach(vData => {
    memberMap.get(vData.owner).vehicles.push(vData);
  });

  // Further processing and rendering of the data
};

As your database grows, your queries will become significantly wasteful. So to account for this, you can elect to show a handful of members at a time instead. Below, the method has been tweaked so that it now fetches 10 members at a time, along with only their payments and vehicles. Note: to increase this number beyond 10, you'll need to break the memberUserIdsForPage array into smaller arrays of up to 10 IDs to account for querying limitations.

const fetchPageOfCustomers = async (prevDocId) => {
  // Fetch handful of member documents from the 'membersCol' collection
  const memberMapForPage = (await getDocs(
    query(membersCol, limitToFirst(10), orderBy(documentId()), prevDocId && startAfter(prevDocId))
  )).docs
    .reduce(
      (map, d) => map.set(
        d.id, // maybe needs to be d.get("uid") if uid <> docId
        { ...d.data(), payments = [], vehicles = [] }
      ),
      new Map()
    );

  const memberUserIdsForPage = [...memberMapForPage.keys()];

  // fetch dependencies together in parallel
  const [paymentsQS, vehiclesQS] = await Promise.all([
    // Fetch payment documents from the 'paymentsCol' collection for the given members
    getDocs(
      query(paymentsCol, where('recipient', 'in', memberUserIdsForPage))
    ),
    // Fetch vehicle documents from the 'vehiclesCol' collection for the given members
    getDocs(
      query(vehiclesCol, where('owner', 'in', memberUserIdsForPage))
    )
  ]);

  paymentsQS.forEach(d => {
    const pData = d.data();
    memberMap.get(pData.recipient).payments.push(pData);
  });

  vehiclesQS.forEach(d => {
    const vData = d.data();
    memberMap.get(vData.owner).vehicles.push(vData);
  });

  return {
    members: memberMapForPage.values(),
    lastDocId: memberUserIdsForPage.pop()
  };
};

Depending on your user interface, it may be desirable to get the members for the current page, display them with spinners for payments and vehicles, then fetch the payments and vehicles, and fill the data into the interface.

Answer 2

There is no singular best way to model data in most NoSQL databases. Instead it all depends on the use-cases of your app, and you'll regularly need to adapt your data model as you add/change use-cases.

Your current data model seems highly normalized. There is nothing wrong with that, but (given that Firestore doesn't support server-side joins) it does mean that you end up loading lots of individual documents. So your write operations are simple and singular, but your load operations get more complex and slow.

The other extreme is if you'd replicate all data you need for the member page into a single document. This affects multiple write operations, as they'll all need to write data to multiple locations - so that's more complex and slower. On the other hand, the read operation will now be much simpler and scaleable, as you need to load a single document.

Any data model you pick will be somewhere between these extremes, and depend on your use-cases and your experience/comfort level around data duplication and fan-out operations.

To learn more about this, I recommend reading NoSQL data modeling and watching Get to know Cloud Firestore.