Auto scaling a Pull Queue in App Engine

Question

I'm trying to implement a Push Notification system in PHP which needs to send massive notifications with the shortest delay possible, as described here:

The drawback I see in this architecture is how to auto scale the Notification Workers. As far as I know, there is no way to count the pending tasks in a pull queue, nor to count the active workers. How would you do it?

score 0 · Answer 1 · answered Nov 04 '16 at 23:53

0

There's queue statistics where you can get the number of tasks in the queue https://cloud.google.com/appengine/docs/python/refdocs/google.appengine.api.taskqueue. We use it for monitoring the number of tasks, seems it can be used for your purpose.

answered Nov 04 '16 at 23:53

marcadian

2,608
13
20

score 0 · Accepted Answer · answered Nov 14 '16 at 14:27

I solved this counting the pending tasks in the Notification Queue with this method, and then adding workers depending on the number of pending tasks:

public static function task_count($queue)
    {
        $request = new google\appengine\TaskQueueFetchQueueStatsRequest();
        $response = new  google\appengine\TaskQueueFetchQueueStatsResponse();

        $request->addQueueName($queue);            

        google\appengine\runtime\ApiProxy::makeSyncCall('taskqueue', 'FetchQueueStats', $request, $response);

        return $response->getQueueStats(0)->getNumTasks();
    }

Auto scaling a Pull Queue in App Engine

2 Answers2