How can I limit the amount of writes that can be done to a certain collection in Cloud Firestore?

Question

I'm looking for a way to prevent writing more than a given limit of documents to a (sub)collection in a given periode.

For example: Messenger A is not allowed to write more then 1000 Messages per 24 hours.

This should be done in the context of an Firebase Function API endpoint because it's called by third parties.

The endpoint

app.post('/message', async function(req:any, res:any) {
        // get the messenger's API key
        const key = req.query.key

        // if no key was provided, return error
        if(!key) {
            res.status(400).send('Please provide a valid key')
            return
        }

        // get the messenger by the provided key
        const getMessengerResult = await admin.firestore().collection('messengers')
            .where('key', '==', key).limit(1).get()

        // if there is no result the messenger is not valid, return error
        if (getMessengerResult.empty){
            res.status(400).send('Please provide a valid key')
            return
        }

        // get the messenger from the result
        const messenger = getMessengerResult.docs[0]

        // TODO: check if messenger hasn't reached limit of 1000 messages per 24 hours

        // get message info from the body
        const title:String = req.body.title
        const body: String = req.body.body

        // add message
        await messenger.ref.collection('messages').add({
            'title':title,
            'body':body,
            'timestamp': Timestamp.now()
        })

        // send response
        res.status(201).send('The notification has been created');
    })

One thing I've tried was the following piece of code in place of the TODO::

// get the limit message and validate its timestamp
const limitMessageResult = await messenger.ref.collection('messages')
    .orderBy('timestamp',"desc").limit(1).offset(1000).get()

if(!limitMessageResult.empty) {
    const limitMessage = limitMessageResult.docs[0]
    const timestamp: Timestamp = limitMessage.data()['timestamp']

    // create a date object for 24 hours ago
    const 24HoursAgo = new Date()
    24HoursAgo.setDate(24HoursAgo.getDate()-1)

    if(24HoursAgo < timestamp.toDate()) {
        res.status(405).send('You\'ve exceeded your messages limit, please try again later!')
        return
    }
}

This code works, but there is a big BUT. The offset does indeed skip the 1000 results, but Firebase will still charge you for it! So every time the messenger tries to add 1 message, 1000+ are read... and that's costly.

So I need a better (cheaper) way to do this.

One thing I've come up with, but haven't yet tried would be adding an index/counter field to a message that increases by 1 every message. Then instead of doing:

const limitMessageResult = await messenger.ref.collection('messages')
    .orderBy('timestamp',"desc").limit(1).offset(1000).get()

I could do something like:

const limitMessageResult = await messenger.ref.collection('messages')
    .where('index','==', currentIndex-1000).limit(1).get()

But I was wondering if that would be a save way. For example, what would happen if there are multiple request at the same time. I would first need to get the current index from the last message and add the new message with index+1. But could two requests read, and thus write the same index? Or could this be handled with transactions?

Or is there a totally different way to solve my problem?

Answer 1

I have a strong aversion against using offset() in my server-side code, precisely because it makes it seem like it's skipping documents, where it's actually reading-and-discarding them.

The simplest way I can think of to implement your maximum-writes-per-day count is to keep a writes-per-day counter for each messenger, that you then update whenever they write a message.

For example, you could do the following whenever you write a message:

await messenger.ref.collection('messages').add({
    'title':title,
    'body':body,
    'timestamp': Timestamp.now()
})
const today = new Date().toISOString().substring(0, 10); // e.g. "2020-04-11"
await messenger.ref.set({
  [today]: admin.firestore.FieldValue.increment(1)
}, { merge: true })

So this adds an additional field to your messenger document for each day, whee it then keeps a count of the number of messages that messenger has written for that day.

You'd then use this count instead of your current limitMessageResult.

const messageCount = (await messenger.get()).data()[today] || 0;
if (messageCount < 1000) {
  ... post the message and increase the counter
}
else {
  ... reject the message, and return a message
}

Steps left to do:

• You'll want to secure write access to the counter fields, as the messenger shouldn't be able to modify these on their own.
• You may want to clean out older message counts periodically, if you're worried about the messenger's document becoming too big. I prefer to leave these types of counters, as they give an opportunity to provide some stats cheaply if needed.