Scheduled AWS Lambda Task at less than 1 minute frequency

Question

We are trying to develop a true lambda-based application in which certain tasks need to be performed at schedules of variable frequencies. They are actually polling for data, and at certain times of the day, this polling can be as slow as once every hour, while at other times, it has to be once every second. I have looked at the options for scheduling (e.g. Using AWS Lambda with Scheduled Events and AWS re:Invent 2015 | (CMP407) Lambda as Cron: Scheduling Invocations in AWS Lambda), but it seems that short of spinning up an EC2 instance or a long-running lambda, there's no built-in way of firing up lambdas at a frequency of less than one minute. The lambda rate expression doesn't have a place for seconds. Is there a way to do this without an EC2 instance or long-running lambda? Ideally, something that can be done without incurring additional cost for scheduling.

Answer 1

You theoretically can wire up a high-frequency task-trigger without an EC2 instance or long-running lambda using an AWS Step Function executed by a CloudWatch Events scheduled event (see Emanuele Menga's blog post for an example), but that approach would actually be more expensive than the other two options, according to my current (as of May 2019) estimates:

(assuming 1 year = 31536000 seconds)

Step Function:

• $0.0250 per 1000 state transitions
• 2 state transitions per tick (Wait, Task) + at least 1 per minute (for setup/configuration)
• 31536000 * (2 + 1/60) * 0.0250 / 1000 = $1589.94/year, or as low as $65.70/year for lower frequency trigger (2 ticks per minute)

Lambda Function:

• $0.000000208 per 100ms (for smallest 128mb function)
• 31536000 * 0.000000208 * 10 = $65.595488/year

EC2 Instance:

• t3a.nano is $0.0047 per hour on-demand, or as low as $0.0014 using Spot instances
• 31536000 * 0.0047 / 3600 = $41.172/year, or $12.264/year using Spot instances

So there will be some scheduling cost, but as you can see the cost is not that much.

Answer 2

Currently lambda functions are invoked, at the very least, every 1 minute from Cloudwatch schedule events.

A solution that might work would be the following:

Setup an EC2 instance and from a program, that you will run as a background job, use the aws sdk to invoke your lambda function. Example:

while True:
  invoke lambda function via aws sdk
  sleep for x seconds

Answer 3

At this point in time AWS Lambda allows functions to be scheduled to run every 5 minutes with a maximum execution time of 5 minutes.

This means if you want to run an AWS Lambda function at intervals less than every 5 minutes while not using EC2 you can take a two phased approach. Create an AWS Lambda function to run every 5 minutes and have it actually run for the entire 5 minutes calling other AWS Lambda functions asynchronously at the appropriate time intervals.

This approach will cost you more since the first AWS Lambda function that runs for the entire 5 minutes will essentially be running continuously, but you can reduce the cost by having the smallest amount of RAM allocated.

---

UPDATE

CloudWatch Events now allow for schedules at a frequency of 1 minute. This means you can now schedule a Lambda function to run every minute and have it run for up to a minute to achieve sub-minute accuracy. It is important to note that scheduled events do not fire at the beginning of every minute so achieving exact sub-minute timing is still a bit tricky.

Answer 4

Don't poll with lambda or you are better off using EC2 if you expect to spend more time polling than performing work. Lambda charges by execution time and polling is costly.

You really need an event driven system with Lambda. What determines when you poll?

Answer 5

There is a simple hack though where you could use a setTimeout or setInterval.

i.e:

    'use strict';
    async function Task() { 
      console.log('Ran at'+new Date())
    }
    const TIMEOUT = 30000;
    exports.handler = (event, context) => {
       return Promise.all([
          Task(), 
          new Promise(function(resolve, reject) {
             let timeout = setTimeout(function() {
               Task().then(() => {
                 clearTimeout(timeout)
               })
             }, TIMEOUT);
          })
          
    ])
        
    }