1

I have a lambda function on the nodejs4.x runtime. If my script stops execution due to an error, lets say I try to get .length of an undefined object, then I can't start the script again. It's not even like the script runs and hits the same error, the script doesn't run. The lambda handler function is never called the second time.

This lambda function is the endpoint for Amazon Alexa. When I reupload the code (a zip file) then the system works again.

Is this some behavior of nodejs? Is the script ending prematurly corrupting the files so it cannot start again?

When the server hits an error I get this message Process exited before completing request

And then subsequent requests hit the timeout limit.

Important Edit I have pinpointed the issue to NPM request. the module doesnt finish loading ie.

console.log('i see this message');
var request = require('request');
console.log('this never happens');
Hurricane Development
  • 2,449
  • 1
  • 19
  • 40

4 Answers4

1

Couple of things that I know:

  1. If lambda invocation fails, due to any reason, it will be invoked again (actually it will be retried at most 3 times). However, this is only true for asynchronous invocations, there are two types of invocations.

  2. Any external module that your lambda's code requires, must be included in the package that you deploy to the lambda, I have explained this simply in here.

You can write code that accesses a property of undefined variable, yes it will throw an exception, and if this invocation is asynchronous it will be retried 2 more times - which will fail too of course.

Community
  • 1
  • 1
johni
  • 5,342
  • 6
  • 42
  • 70
0

Since the Lambda function fails when calling require('request') I believe that the project has not been deployed correctly. request must be deployed with the Lambda function because it is not part of Node.js 4.3.2 (the current Lambda JavaScript runtime).

Make sure that:

  1. require is added to your package.json file (e.g. by calling $ npm install require --save, see npm install for details).
  2. You create a deployment package by zipping your project folder (including the node_modules folder).
  3. That the deployment .zip is uploaded to your Lambda function.
matsev
  • 32,104
  • 16
  • 121
  • 156
0

So after contacting AWS through their forums, this turns out to be a bug. The container is not cleared upon an error, so the code has to be re-uploaded.

A solution is to make a cloudwatch alarm that fires another lambda function that re uploads the code automatically.

They are working on a fix.

Forum post: https://forums.aws.amazon.com/thread.jspa?threadID=238434&tstart=0

Hurricane Development
  • 2,449
  • 1
  • 19
  • 40
0

In fact there are many cases when Lambda becomes unresponsive, e.g.:

  1. Parsing not valid json:

    exports.handler = function(event, context, callback)
{
    var nonValidJson = "Not even Json";
    var jsonParse = JSON.parse(nonValidJson);

  2. Accessing property of undefined variable:

    exports.handler = function(event, context, callback)
{
    var emptyObject = {};
    var value = emptyObject.Item.Key;

  3. Not closing mySql connection after accessing RDS leads to Lambda timeout and then it becomes non-responsive.

Making a lambda that reuploads the code can take a portion of time. After some tests it's revealed that in fact Lambda tries to restart (reload the container?), there is just not enough time. If you set the timeout to be 10s, after ~4s of execution time Lambda starts working, and then in next runs comes to behave normally. I've also tried playing with setting:

context.callbackWaitsForEmptyEventLoop = false;

and putting all 'require' blocks inside handler, nothing really worked. So the good way to prevent Lambda becoming dead is setting bigger timeout, 10s should be more than enough as a workaround protection against this bug.

Slava Elantsev
  • 81
  • 1
  • 6