Flatlist appending same data to the array during pagination?

Question

We have a flatlist of "thoughts", and are getting the first 10 items, followed by the next 7 every page after.

However, when you reach the end of the first "page", the same data gets appended to the array, and you scroll through the same page over and over

Our flatlist:

<FlatList
      ref={scrollRef}
      data={thoughts}
      renderItem={renderItem}
      keyExtractor={item => item.key}
      contentContainerStyle={{ marginTop: 45, paddingBottom: 50 }}
      showsHorizontalScrollIndicator={false}
      showsVerticalScrollIndicator={false}
      onRefresh={refresh}
      refreshing={isLoading}
      onEndReached={getMore} // <------- Get the next page of posts
    />

Our "getMore" - we first convert the date_created from an object to a firebase timestamp, because we are sorting by date when we call firebase.

const getMore = async() => {
  const lastItemIndex = thoughts.length - 1;
  const seconds = thoughts[lastItemIndex].date_created._seconds;
  const nanoseconds = thoughts[lastItemIndex].date_created._nanoseconds;
  const lastTime = new firebase.firestore.Timestamp(seconds, nanoseconds); //-- the firebase timestamp

  const getMoreThoughtsOneCategory = firebase.functions().httpsCallable('getMoreThoughtsOneCategory');
  getMoreThoughtsOneCategory({
    index: lastItemIndex,
    category: selectedCategory,
    date_created: lastTime.toDate(),
  }).then((result) => {
    setThoughts(thoughts.concat(result.data)); //Append the collection
  }).catch((err) => {
    console.log(err);
});

};

Our getMoreThoughtsOneCategory firebase cloud function:

exports.getMoreThoughtsOneCategory = functions.https.onCall((data, context) => {
    return new Promise((resolve, reject) => {

                admin.firestore()
                .collection('thoughts')
                .where("category", "==", data.category) 
                .orderBy('date_created', 'desc') 
                .startAfter(data.date_created)
                .limit(7)
                .get()
                .then((query) => {
                    const thoughts = [];
                    query.forEach((res) => {
                      const {
                        ...Get fields
                      } = res.data();
          
                      thoughts.push({
                        ...Set fields
                      });
                  });
                  resolve(thoughts);
                  return null;
    
                }).catch(err => {
                    console.log("Errors from getting more thoughts " + err);
                    reject(err);
                })

    });
})

And here is the initial query for the first "page" of thoughts, the data that keeps repeating:

exports.getThoughtsOneCategory = functions.https.onCall((data, context) => {
    return new Promise((resolve, reject) => {
        admin
        .firestore()
        .collection('thoughts')
        .where("category", "==", data.category)
        .orderBy('date_created', 'desc')
        .limit(9)
        .get()
        .then((query) => {
          const thoughts = [];
          query.forEach((res) => {
            const {
                ... Get fields
            } = res.data();

            thoughts.push({
              ... Set fields
            });

            index++;
          });
          resolve(thoughts);
          return null;

        }).catch(err => {
            console.log("Error from getting first set of thoughts " + err);
            reject(err);
        })
  
    });
});

To summarize, we get the first 9 items. When we reach the end of the flatlist, we should get the next 7 items in the db, but we just get the first 7 items appended to the array. How can we fix this issue?

Answer 1

Problem 1) Promise Constructor Antipatterns

To begin, the Firebase SDKs are Promise-based APIs, so you should eliminate the Promise Constructor Antipatterns in your code. I recommend reading up on Promises and how they work, paying attention how to migrate older code. Once you've read that, take a look at this thread covering the antipattern.

In its simplest form, a PCAP looks like this:

return new Promise((resolve, reject) => {
  somePromise
    .then((result) => {
      // do some work
      resolve(newResult);
    })
    .catch((err) => reject(err));
})

The above code introduces a whole bunch of weird quirks and doesn't properly chain the promises together. The functional equivalent of the above would be:

return somePromise
  .then((result) => {
    // do some work
    return newResult;
  })

Applying these changes to your functions:

exports.getThoughtsOneCategory = functions.https.onCall((data, context) => {
  return admin.firestore()
    .collection('thoughts')
    .where("category", "==", data.category)
    .orderBy('date_created', 'desc')
    .limit(9)
    .get()
    .then((querySnapshot) => {
      const thoughts = [];
      querySnapshot.forEach((res) => {
        const {
            ... Get fields
        } = res.data();

        thoughts.push({
          ... Set fields
        });
      });
      return thoughts;
    })
    .catch(err => {
      console.error("Error from getting first set of thoughts ", err);
      // rethrow errors for client
      throw new functions.https.HttpsError('unknown', err.message);
    });
});

exports.getMoreThoughtsOneCategory = functions.https.onCall((data, context) => {
  return admin.firestore()
    .collection('thoughts')
    .where("category", "==", data.category) 
    .orderBy('date_created', 'desc')
    .startAfter(data.date_created)
    .limit(7)
    .get()
    .then((query) => {
      const thoughts = [];
      query.forEach((res) => {
        const {
          ...Get fields
        } = res.data();

        thoughts.push({
          ...Set fields
        });
      });
      return thoughts;
    })
    .catch(err => {
      console.error("Error from getting more thoughts ", err);
      // rethrow errors for client
      throw new functions.https.HttpsError('unknown', err.message);
    });
});

Note: I refactoring those functions to use the same code internally as they share a lot of the same code and you want to ensure your thoughts arrays are processed and built the same way.

Problem 2) Parameter serialization

When you call your callable function in your client code:

getMoreThoughtsOneCategory({
  index: lastItemIndex,
  category: selectedCategory,
  date_created: lastTime.toDate(),
})

You attempt to send the following data:

{
  index: number,
  category: string,
  date_created: Date
}

The problem here is that date_created is not one of the standard JSON types and what you end up sending is:

{
  index: number,
  category: string,
  date_created: {}
}

This leads to your getMoreThoughtsOneCategory function incorrectly thinking it has a Date object for data.date_created. Because data.date_created isn't undefined, .startAfter(data.date_created) doesn't throw an error and just tries to use the incorrect value. When Firestore receives the query, it sees an empty startAfter parameter and just returns the first 7 entries that match the rest of the query.

From the MDN page on JSON:

JSON can represent numbers, booleans, strings, null, arrays (ordered sequences of values), and objects (string-value mappings) made up of these values (or of other arrays and objects). JSON does not natively represent more complex data types like functions, regular expressions, dates, and so on. (Date objects by default serialize to a string containing the date in ISO format, so the information isn't completely lost.) If you need JSON to represent additional data types, transform values as they are serialized or before they are deserialized.

So, because Date isn't supported, we need to change the representation of date_created to a compatible type. We have three options:

• Timestamp#toJSON() will give a plain object with the shape { seconds: number, nanoseconds: number }, it's unlikely to change but undocumented.
• Timestamp#valueOf() will give a string representation of the timestamp as "<seconds>.<nanoseconds>".
• Date#toISOString() will convert the Date to an ISO 8601 standards-compliant string
• Date#getTime() or Timestamp#toMillis() will convert the timestamp to the number of milliseconds since 1970-01-01 UTC.

Of these options, using the string or number are the simplest approaches. Because Date#toISOString() requires us to convert to a Date first, we'll just use Timestamp#toMillis(). However, by using milliseconds this way, you lose the precision of the Timestamp class. So we'll make use of Timestamp#valueOf() like so:

getMoreThoughtsOneCategory({
  index: lastItemIndex,
  category: selectedCategory,
  date_created: lastTime.valueOf(),
})

Then in getMoreThoughtsOneCategory, tweak it to use this value using either:

.startAfter(data.date_created) // Timestamp string form

or

// restore Timestamp object from Timestamp.valueOf() form
const dateCreatedTimestamp = new admin.firestore.Timestamp(...(data.date_created.split(".").map(Number));
/* ... */
.startAfter(dateCreatedTimestamp) // Timestamp object form

To summarize, don't assume what you send the server in the arguments of a httpsCallable will look the exactly same at the other end. You should check what you are receiving, ideally using some form of error handling.

Answer 2

First and foremost I’d be sure to skip the number of items you already have, or go to whatever index you left off at when it comes to the DB fetch. Beyond that, I’d double check how you’re rendering the flat list, or, more specifically, if your causing a re-render in, say, a useEffect call or similar.