20

I'm having problem with a Lambda function which for some reason it goes on timeout.

I tried to add some console.log to understand what was going on and everything works properly until the external http request.

I use the npm module request

I can see the log before the request, but the one inside the callback never shows up, like if the endpoint doesn't respond.

Am I missing something? Thanks in advance!

var mysql = require('mysql'); 
var request = require('request');
var config = require('./config.json'); 

var pool = mysql.createPool({
  host: config.dbhost,
  user: config.dbuser,
  password: config.dbpassword,
  database: config.dbname
});

var openWeatherOptions = (device) => {
    let domain = 'https://api.openweathermap.org/data/2.5/';
    let apiEndpoint = 'weather?';
    let params = `lat=${device.lat}&lon=${device.lng}&APPID=${process.env.WEATHER_APP_ID}&units=metric`;
    let url = [domain, apiEndpoint, params].join('').toString();
    return {
      url: url,
      method: 'GET',
      headers: { 'Content-Type': 'application/json'},
    };
};

exports.handler = (event, context, callback) => {
    context.callbackWaitsForEmptyEventLoop = false;
    // Connect to DB
    console.log('fallback connecting to DB')
    pool.getConnection( (errOnConnection, connection) => {
        if(errOnConnection) {
            callback({"err": errOnConnection });
        }
        console.log('fallback connected to DB')
        // Retrieve device position
        let queryOne = 'SELECT lat, lng FROM T03_DevicesPosition WHERE deviceCode = (?);';
        console.log('fallback retrieving device position')
        connection.query( queryOne, [event.device], (errOnQ1, results, fields) => { 
            if(errOnQ1) {
                connection.release();
                callback({"err": errOnQ1 });
            }
            console.log('fallback device position retrieved')

            // Call openweather 
            let device = results[0];
            let options = openWeatherOptions(device);
            console.log('fallback calling openWeather with following data: ', device, options);

            request( options, (errOpenWeather, response, body) => {
                console.log('fallback openweather response received');
                if(errOpenWeather || (response.statusCode !== 200 && response.statusCode !== 201) ) {
                    connection.release();
                    callback({"err": errOpenWeather });
                }
                let meteo = JSON.parse(body).main;
                meteo.date = new Date(event.time*1000);
                meteo.pressure = Math.floor( meteo.pressure );
                console.log('fallback storing data', meteo);
                let query = `INSERT INTO T02_DevicesTransmissions (deviceCode, transmissionDate, temperature, humidity, pressure, unixDate, rawData) VALUES ( ?, ?, ?, ?, ?, ?, ?);`;
                let queryValues = [ event.device, meteo.date, meteo.temp, meteo.humidity, meteo.pressure, event.time, 'fallback']; 
                connection.query( query, queryValues, (errInsert, results, fields) => { 
                    connection.release();
                    console.log('fallback completed with', errInsert ? '' : 'out');
                    if (errInsert) callback({"err": errInsert });
                    else callback();
                });
            });

        });
    });
}
Robdll
  • 5,865
  • 7
  • 31
  • 51
  • 1
    Here are the AWS instructions for this: https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/ – Michael Hays Oct 20 '21 at 21:29

4 Answers4

40

This is an old question but I spent a few hours banging my head figuring out how to set this up properly, so here's hoping someone else's time will be saved :)

If your lambdas are in a VPC they need to go through a NAT Gateway in order to reach external services.

One way to achieve this is:

  • configure your lambda to use a (or multiple) specific subnets (the Lambda console will suggest you associate at least 2 subnets in different availability zones to your lambda to ensure availability)
  • create a NAT Gateway in a different subnet
  • have the route table associated with your lambda's subnets send all outbound traffic (0.0.0.0/0) to the NAT Gateway you created (you do that by choosing the NAT in the target field)
  • have the route table in the NAT's subnet send all outbound traffic to an Internet Gateway

This will route traffic from your lambdas properly through the NAT and Internet Gateway so they can reach external services.

Note that if you're also using RDS from your lambdas AND you intend to also connect to RDS from outside (e.g. to manage the DB from your local machine), you can't put the RDS instance in the lambdas' subnet, or you won't be able to connect to it from outside (the NAT is outbound only). In that case you should make sure your RDS instance is associated with a subnet that is accessible, for instance the one the NAT is in (i.e. the one that sends outbound traffic to the Internet Gateway).

Jules Olléon
  • 6,733
  • 6
  • 37
  • 47
  • 1
    Yep that's the correct answer, I don't remember how I ended up understanding it but I won't forget the struggle of reading pages and pages of doc. Also, I remember it was important to provide to the lambda configuration one or more subnet (can't remember if they need to be private, public or doesn't make different) – Robdll Apr 17 '20 at 13:50
  • 1
    Yeah, AWS networking takes some time to get used to... you do need to associate subnet(s) to the lambda in its VPC config (AWS recommends at least 2 in different availability zones, and it picks one each time the lambda is instanciated). The important part is that the route table for these subnets needs to send outbound traffic to the NAT (which itself needs to have internet access). – Jules Olléon Apr 18 '20 at 20:36
  • 1
    THANKYOU SO MUCH I've spent literally 6+ hours googling and trying every code variation of a nodejs http request I could think off whilst tearing my hair out. This worked and saved me. Thanks for documenting. – phonicx Jul 09 '20 at 10:03
  • NAT gateway cost ~$30/month. if you can make vpc peering connection from your external service, it will be free. both use pricing for data transfer though. ~0.04/gb for nat and 0.01 for peering, at least today in n.virginia region – Lukas Liesis Apr 03 '21 at 14:25
  • @LukasLiesis would u be link any resources where I can read up on it? – Patryk Szylin Dec 16 '21 at 16:44
  • @pejno on AWS docs. AWS has excellent coverage in docs about close to every single thing. They have html docs like this url, also pdf, videos on youtube, aws scholar program, google. For e.g. NAT: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html – Lukas Liesis Dec 17 '21 at 07:41
  • @LukasLiesis I agree AWS is pretty good for it. But I'm specifically interested on finding out how to setup an external service via VPC peering – Patryk Szylin Dec 17 '21 at 10:19
  • @pejno check https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html there is PDF with all-in-one-place: https://docs.aws.amazon.com/vpc/latest/peering/vpc-pg.pdf and all VPC docs are here: https://docs.aws.amazon.com/vpc/index.html I would suggest to start from watching re:invent videos about networking on youtube to get the overview if you are early in this – Lukas Liesis Dec 18 '21 at 12:52
4

You don't have a VPC set, right? And if you do, have to check that you have a NAT gateway attached if you're on a private subnet.

noah haibach
  • 81
  • 1
  • 4
  • I do have a VPC, and I have a couple of lambda functions running on it. This is the only one that communicate externally however. – Robdll Apr 30 '18 at 18:42
  • 1
    I think your answer pointed me on the right way, I'm reading this one: https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/ – Robdll Apr 30 '18 at 18:44
3

This is your solution, please read.

https://forums.developer.amazon.com/questions/95692/invoking-a-rest-api-from-within-a-lambda-function.html

Further

This code sample shows how to call and receive external rest service data, within your skill Lambda code. https://github.com/robm26/SkillsDataAccess/blob/master/src/CallService/index.js

Ritesh Aryal
  • 825
  • 9
  • 13
0

Jules Olléon has put it nicely. Remember to put the subnet of your lambda VPC private. it won't work with public subnet.

you can read more in details in the AWS Support here

mdrijwan
  • 41
  • 1
  • 8