AWS Lambda with Knex JS and RDS Postgres

Question

I have been doing some research and I'm not able to find a good answer about using Knex JS within a Lambda function:

How do I use Knex with AWS Lambda? #1875

Serverless URL Shortener with Apex and AWS Lambda

Here is what I have in my index.js:

const knex = require('knex')({
  client: 'pg',
  connection: {...},
});

exports.handler = (event, context, callback) => {
  console.log('event received: ', event);
  console.log('knex connection: ', knex);

  knex('goals')
    .then((goals) => {
      console.log('received goals: ', goals);
      knex.client.destroy();
      return callback(null, goals);
    })
    .catch((err) => {
      console.log('error occurred: ', err);
      knex.client.destroy();
      return callback(err);
    });
};

I am able to connect and execute my code fine locally, but I'm running into an interesting error when it's deployed to AWS - the first call is always successful, but anything after fails. I think this is related to the knex client being destroyed, but then trying to be used again on the next call. If I re-upload my index.js, it goes back to working for one call, and then failing.

I believe this can be resolved somehow using promises but this my first time working with Lambda so I'm not familiar with how it's managing the connection to RDS on subsequent calls. Thanks in advance for any suggestions!

Hongbo Miao · Answer 1 · 2020-12-03T14:05:24.470

For me, it worked on my local machine but not after deploying. I was kind of be mislead.

It turns out the RDS inbound source is not open to my Lambda function. Found solution at AWS Lambda can't connect to RDS instance, but I can locally?: either changing RDS inbound source to 0.0.0.0/0 or use VPC.

After updating RDS inbound source, I can use Lambda with Knex successfully.

The Lambda runtime I am using is Node.js 8.10 with packages:

knex: 0.17.0
pg: 7.11.0

The code below using async also just works for me

const Knex = require('knex');

const pg = Knex({ ... });

module.exports.submitForm = async (event) => {
  const {
    fields,
  } = event['body-json'] || {};

  return pg('surveys')
    .insert(fields)
    .then(() => {
      return {
        status: 200
      };
    })
    .catch(err => {
      return {
        status: 500
      };
    });
};

Hopefully it will help people who might meet same issue in future.

score 1 · Answer 2 · answered Jan 16 '18 at 23:03

The most reliable way of handling database connections in AWS Lambda is to connect and disconnect from the database within the invocation process itself.

In your code above, since you disconnected already after the first invocation, the second one does not have a connection anymore.

To fix it, just move your instantiation of knex.

exports.handler = (event, context, callback) => {
  console.log('event received: ', event);

  // Connect
  const knex = require('knex')({
    client: 'pg',
    connection: {...},
  });

  console.log('knex connection: ', knex);

  knex('goals')
    .then((goals) => {
      console.log('received goals: ', goals);
      knex.client.destroy();
      return callback(null, goals);
    })
    .catch((err) => {
      console.log('error occurred: ', err);
      // Disconnect
      knex.client.destroy();
      return callback(err);
    });
};

There are ways to reuse an existing connection but success rates for that varies widely depending on database server configuration and production load.

In case of high rate of execution of lambda functions, you will hit connections limit. Have a look at - https://dba.stackexchange.com/q/161524/88470 — Adarsh Madrecha, Sep 15 '19 at 06:01
@AdarshMadrecha This answer connects and disconnects in each invocation so it would be unlikely to reach the connections limit. — Noel Llevares, Sep 15 '19 at 22:24
@NoelLlevares Can you elaborate why you think this? Lambdas will automatically scale under heavy loads and can result in many more db connections being created (or being attempted to be created) than the DB can handle. — Eugene Kim, Oct 03 '20 at 18:44

score 0 · Answer 3 · answered Jan 07 '21 at 12:36

I got the exact same issue as you said: Used destroy() in an AWS lambda function (like this: await knex.destroy() at the bottom) and suddenly all my AWS lambdas were in error.

Because I did not suspect it, I searched for hours what was causing the issue and even started to investigate using lambda + vpc + nat etc.. Turns out it's just that AWS freezes lambda in a way that if you destroy the connection, on the next handler invocation it will try to reuse the connection.

Solution: do not use .destroy() at the end of lambda and redeploy.

AWS Lambda with Knex JS and RDS Postgres

3 Answers3

Linked