3

I am using context.logStreamName to identify the lambda instance. Concurrency is set to unreserved. But it the log shows out only 50 instance is working.

Am I misunderstand the logStream (pre logStream pre instance)? I'd got the information from this blog

Below is my corresponding code: 

let instances = {};
for (let i = 0; i < each_invokes; i++) {
    lambda.invoke(params, function (err, data) {
        if (err) {
            console.log(err, err.stack);
        } else {
            // console.log(data);
            let logs = Buffer.from(data.LogResult, 'base64').toString('utf8');
            let instanceID = logs.match(new RegExp("\\$InstanceID: (.*?) \\$END", "ig"))[0].split(" ")[1];
            if (instanceID) {
                if (instances.hasOwnProperty(instanceID)) {
                    instances[instanceID] += 1
                } else {
                    instances[instanceID] = 0
                }
                console.log(instanceID, instances[instanceID], 'StatusCode:', data.StatusCode);
            } else {
                console.log('missing instanceID:', data.StatusCode)
            }
        }
        console.log(Object.keys(instances).length)
    });
}

I console.log(context.logStreamName) on my lambda function and use RegExp to warp out instance ID.

The tail log keeps showing the below output:

b4afda96edf04e07ab819589d298521d 3 StatusCode: 200
50
a1ef1f4b518d47398daedda434ddd48c 3 StatusCode: 200
50
8dedc647796545ada94770895a8b190a 3 StatusCode: 200
50
7c4f48d72de0486e86e47c76ed3269ec 3 StatusCode: 200
50
2aa1f0dc9b0440fcac3d3127d123edb8 3 StatusCode: 200
50
3d1abeb38e7a45a4b933b669ad0c12d7 3 StatusCode: 200
50
531883ce26ac43b9bb0976726a2e5aa6 3 StatusCode: 200
50
cb9d0611a2e24244bd9ee98967a768c2 3 StatusCode: 200
50

So is it means that, the AWS lambda can only run 50 instance? If it does, how can I increase the count of concurrency in AWS Lambda?

Matus Dubrava
  • 13,637
  • 2
  • 38
  • 54
Bruce
  • 115
  • 1
  • 8
  • I've looked at your code some more. It seems the above code is incomplete (need `lambda`, `each_invokes` and `params`). The code above seems to be the invoking side code. We also need the lambda side code. We are happy to solve this, please post the minimum code we need to reproduce this problem: https://stackoverflow.com/help/minimal-reproducible-example – Taterhead Jun 14 '19 at 11:34
  • @Taterhead `lambda` is a client stated by [this doc](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html#invoke-property); `each_invokes` is the concurrency request times, I set it to 100; params can also found by the doc mentioned before. – Bruce Jun 17 '19 at 02:35
  • Hey @Bruce, the code listed above is half of the story. We need to see the code listed for your Lambda function also. The code running on AWS. We just need the minimum code so we can reproduce it getting throttled at 50. – Taterhead Jun 17 '19 at 07:31

3 Answers3

5

Apart from that 1000 number of concurrent executions soft limit, there are other things to consider.

If you are using lambda functions to process events that are not poll based then the actual number of functions that you can run at a single moment is not 1000. The formula to calculate the number is

invocations per second * average execution duration in seconds

Which means that if your function takes 20 seconds on average to execute then the number of instances of that function that you can run simultaneously is 1000 / 20 which is equal to 50.

I am not saying that this is exactly the case with your code, but it is definitely something to consider when working with lambda limits.

If we are talking about poll based functions (Lambda integrated with Kinesis streams, DynamoDB streams) then another thing that comes into play is the number of shards in the stream. Each shard in the stream can be processes by at most 1 instance of a particular lambda function at a time (note that different lambda functions can process the same shard simultaneously). So if you have 50 shards in your stream then your lambda function can scale up to 50 instances of it, but the formula mentioned above still holds so if your function takes more than 20 seconds to process the message then the number of concurrent executions will be less than 50.

Matus Dubrava
  • 13,637
  • 2
  • 38
  • 54
3

I think one of the reason why I got 50 instance log is the balance between latency time of last lambda function process and the time of next cold start. The configuration setting of Concurrency does not tell me it will cold start up 1000 unreserved account concurrency instantly. It do receive 1000 requests but once it will immediately free for next request, so there is no need to cold start another instance. But I am not sure whether it is true or not.

The other reason that I can confirm is the default limit of AWS.

"AWS Lambda has a default safety throttle of 100 concurrent executions per account per region. If you wish to submit a request to increase the throttle of 100 concurrent executions you can visit our Support Center..."

I did't not find the official doc but find the limit manually:

When I set concurrency to 100, then start two invoke scripts on local machine, each one of scripts send 500 invocation simultaneously. It will receive result log on a queue. The instances number on log will gradually increase to 100 instance, not 50 instance, which is the result by one invoke script.

But what if the concurrency is 100 but I start three invocation scripts? Will it startup 150 instances? No, it will throw out TooManyRequestsException error.

This test is not rigorous, still under investigating. Though not serious problem on producation, I really want to understand why this problem casued.

--UPDATE--

Q: Is there a limit to the number of AWS Lambda functions I can execute at once?

No. AWS Lambda is designed to run many instances of your functions in parallel. However, AWS Lambda has a default safety throttle of 100 concurrent executions per account per region. If you wish to submit a request to increase the throttle of 100 concurrent executions you can visit our Support Center, click “Open a new case”, and file a service limit increase request.

Q: What happens if my account exceeds the default throttle limit on concurrent executions?

On exceeding the throttle limit, AWS Lambda functions being invoked synchronously will return a throttling error (429 error code). Lambda functions being invoked asynchronously can absorb reasonable bursts of traffic for approximately 15-30 minutes, after which incoming events will be rejected as throttled. In case the Lambda function is being invoked in response to Amazon S3 events, events rejected by AWS Lambda may be retained and retried by S3 for 24 hours. Events from Amazon Kinesis streams and Amazon DynamoDB streams are retried until the Lambda function succeeds or the data expires. Amazon Kinesis and Amazon DynamoDB Streams retain data for 24 hours.

---UPDATE---

As it mentioned by me and Matus Dubrava, I managed my lambda function complexity to monitor the runtime of function. The metrics shows below:

Clearly we can see there is a relationship between Duration(300k Max) and ConcurrentExecutions(I set the max concurreny to 100).

pic

Community
  • 1
  • 1
Bruce
  • 115
  • 1
  • 8
2

As of June 2019 Lambda concurrent executions are set to 1000.

AWS imposes service limits on its resources for various reasons. Most of the limits can be lifted or raised with a service limit increase on your console.

Lambda has a special page describing its limits and an explanation on how to understand how Lambda scales. Only concurrency and Function and Layer Storage can be increased.

We need more information from your problem to determine why you are limited to 50.

If I were to take guess, I'm thinking you are accessing a CloudWatch Log via GetMetricData, which has a limit of 50 per second?

Taterhead
  • 5,763
  • 4
  • 31
  • 40
  • The `context.logStreamName` seems related to CloudWatch, but I am not using GetMetricData. I'm following [official doc](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html#invoke-property) to access the tail log from `aws-sdk` – Bruce Jun 12 '19 at 10:25
  • Hey Bruce, can you update your question with more details on exactly how your Lambda is invoked or triggered? This might provide clues on the 50 limit. Thanks! – Taterhead Jun 12 '19 at 10:33
  • Hi @Taterhead. My lambda function is just like the [example function](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html). I'd post my invoke function on my question. Now I find another problem that if I set concurrent below 100(e.g. 10), once I send over 10 request to lambda. It will throw out `TooManyRequestsException` [error](https://stackoverflow.com/questions/36826352/aws-lambda-toomanyrequestsexception-rate-exceeded). I am not sure whether it is a default limit for my account or just my mistake. Still investigating. Thanks! – Bruce Jun 13 '19 at 03:07
  • 1
    ok @Bruce, I have another suggestion: remove all concurrent limits from your function. Then refer to this page: https://docs.aws.amazon.com/lambda/latest/dg/monitoring-functions-metrics.html - Cloudwatch is the official logging service for AWS and will tell you the true number of concurrent lambdas running. I will try and write up an answer that shows you how to log the number of concurrent lambdas running and see the true number on the dashboard. My hunch is that even though your log says 50, you are actually running 1000. Stay tuned. – Taterhead Jun 13 '19 at 05:40