Node: Check a Firebase db and execute a function when an objects time matches the current time

Question

Background

I have a Node and React based application. I'm using Firebase for my storage and database. In my application users can fill out a form where they upload an image and select a time for the image to be added to their website. I save each image update as an object in my Firebase database like so. Images are arranged in order of ascending update time.

user-name: {
   images: [
     {
        src: 'image-src-url',
        updateTime: 1503953587727
     }
     {
        src: 'image-src-url',
        updateTime: 1503958424838
     }
   ]
}

Scale

My applications db could potentially get very large with a lot of users and images. I'd like to ensure scalability.

Issue

How do I check when a specific image objects time has been met then execute a function? (I do not need assistance on the actual function that is being run just the checking of the db for a specific time.)

Attempts

I've thought about doing a cron job using node-cron that checks the entire database every 60s (users can only specify the minute the image will update, not the seconds.) Then if it finds a matching updateTime and executes my function. My concern is at a large scale that cron job will take a while to search the db and potentially miss a time.

I've also thought about when the user schedules a new update then dynamically create a specific cron job for that time. I'm unsure how to accomplish this.

Any other methods that may work? Are my concerns about node-cron not valid?

Answer 1

There are two approaches I can think of:

1. Keep track of the last timestamp you processed
2. Keep the "things to process" in a queue

Keep track of the last timestamp you processed

When you process items, you use the current timestamp as the cut-off point for your query. Something like:

var now = Date.now();
var query = ref.orderByChild("updateTime").endAt(now)

Now make sure to store this now somewhere (i.e. in your database) so that you can re-use it next time to retrieve the next batch of items:

var previous = ... previous value of now
var now = Date.now();
var query = ref.orderByChild("updateTime").startAt(previous).endAt(now);

With this you're only processing a single slice at a time. The only tricky bit is that somebody might insert a new node with an updateTime that you've already processed. If this is a concern for your use-case, you can prevent them from doing so with a validation rule on updateTime:

".validate": "newData.val() >= root.child('lastProcessed').val()"

As you add more items to the database, you will indeed be querying more items. So there is a scalability limit to this approach, but this approach should work well for anything up to a few hundreds of thousands of nodes (I haven't tested in a while so ymmv).

For a few previous questions on list size:

• Firebase Performance: How many children per node?
• Firebase Scalability Limit
• How many records / rows / nodes is alot in firebase?

Keep the "things to process" in a queue

An alternative approach is to keep a queue of items that still need to be processed. So the clients add the items that they want processed to the queue with an updateTime of when they want to processed. And your server picks the items from the queue, performs the necessary updates, and removes the item from the queue:

var now = Date.now();
var query = ref.orderByChild("updateTime").endAt(now)
query.once("value").then(function(snapshot) {
  snapshot.forEach(function(child) {

    // TODO: process the child node

    // remove the child node from the queue
    child.ref.remove();
  });
})

The difference with the earlier approach is that a queue's stable state is going to be empty (or at least quite small), so your queries will run against a much smaller list. That's also why you won't need to keep track of the last timestamp you processed: any item in the queue up to now is eligible for processing.