2

I am trying to connect MongoDB Atlas with mongoose and aws lambda but i get error MongoNetworkError

  • AWS Lambda
  • Mongoose
  • MongoDB Atlas

The same code was tested with serverless-offline and works perfect, the problem is when i deploy it to AWS Lambda.

This is the code snipet

'use strict';
const mongoose = require('mongoose');
const MongoClient = require('mongodb').MongoClient;
let dbuser  = process.env.DB_USER;
let dbpass = process.env.DB_PASSWORD;
let opts = { 
    bufferCommands: false, 
    bufferMaxEntries: 0, 
    socketTimeoutMS: 2000000, 
    keepAlive: true, 
    reconnectTries: 30, 
    reconnectInterval: 500,
    poolSize: 10,
    ssl: true,
 };
const uri = `mongodb+srv://${dbuser}:${dbpass}@carpoolingcluster0-bw91o.mongodb.net/awsmongotest?retryWrites=true&w=majority`;
// simple hello test
module.exports.hello = async (event, context, callback) => {
    const response = {
        body: JSON.stringify({message:'AWS Testing :: '+ `${dbuser} and ${dbpass}`}),
    };
    return response;
};
// connect using mongoose
module.exports.cn1 = async (event, context, callback)  => {
    context.callbackWaitsForEmptyEventLoop = false;
    let conn = await mongoose.createConnection(uri, opts);
    const M = conn.models.Test || conn.model('Test', new mongoose.Schema({ name: String }));
    const doc = await M.find();
    const response = {
        body: JSON.stringify({data:doc}),
    };
    return response;
};
// connect using mongodb
module.exports.cn2 = (event, context, callback) => {
    context.callbackWaitsForEmptyEventLoop = false;
    console.log("Connec to mongo using connectmongo ");
    MongoClient.connect(uri).then(client => {
        console.log("Success connect to mongo DB::::");
        client.db('awsmongotest').collection('tests').find({}).toArray()
            .then((result)=>{
                let response = {
                    body: JSON.stringify({data:result}),
                }
                callback(null, response)
            })
    }).catch(err => {
        console.log('=> an error occurred: ', err);
        callback(err);
    });
};

In the CloudWatch logs i see this error

{
    "errorType": "MongoNetworkError",
    "errorMessage": "failed to connect to server [carpoolingcluster0-shard-00-02-bw91o.mongodb.net:27017] on first connect [MongoNetworkError: connection 5 to carpoolingcluster0-shard-00-02-bw91o.mongodb.net:27017 closed]",
    "stack": [
        "MongoNetworkError: failed to connect to server [carpoolingcluster0-shard-00-02-bw91o.mongodb.net:27017] on first connect [MongoNetworkError: connection 5 to carpoolingcluster0-shard-00-02-bw91o.mongodb.net:27017 closed]",
        "    at Pool.<anonymous> (/var/task/node_modules/mongodb-core/lib/topologies/server.js:431:11)",
        "    at Pool.emit (events.js:189:13)",
        "    at connect (/var/task/node_modules/mongodb-core/lib/connection/pool.js:557:14)",
        "    at callback (/var/task/node_modules/mongodb-core/lib/connection/connect.js:109:5)",
        "    at runCommand (/var/task/node_modules/mongodb-core/lib/connection/connect.js:129:7)",
        "    at Connection.errorHandler (/var/task/node_modules/mongodb-core/lib/connection/connect.js:321:5)",
        "    at Object.onceWrapper (events.js:277:13)",
        "    at Connection.emit (events.js:189:13)",
        "    at TLSSocket.<anonymous> (/var/task/node_modules/mongodb-core/lib/connection/connection.js:350:12)",
        "    at Object.onceWrapper (events.js:277:13)",
        "    at TLSSocket.emit (events.js:189:13)",
        "    at _handle.close (net.js:597:12)",
        "    at TCP.done (_tls_wrap.js:388:7)"
    ],
    "name": "MongoNetworkError",
    "errorLabels": [
        "TransientTransactionError"
    ]
}

Here is example on github to reproduce the error.

https://github.com/rollrodrig/error-aws-mongo-atlas

Just clone it, npm install, add your mongo atlas user, password and push to AWS.

Thanks.

Imran
  • 5,542
  • 3
  • 23
  • 46
Rolly
  • 3,205
  • 3
  • 26
  • 35
  • Is your mongodb in vpc? – Sailesh Kotha Jul 08 '19 at 16:22
  • No, it is on Mongo Atlas. – Rolly Jul 08 '19 at 16:23
  • I think aws lambda has default 0.0.0.0 egress security group, so perhaps you need to allow the ingress from the carpoolingcluster0? also, not sure lambdas are VPC associated off top of my head, but you might want to take a look at https://www.mongodb.com/blog/post/introducing-vpc-peering-for-mongodb-atlas – 4m1r Jul 08 '19 at 16:30
  • I took your project and able to connect to my Atlas successfully with `0.0.0.0/0` whitelisting. Your error is most likely IP whitelisting cause. You either need to use [NAT](https://medium.com/@matthewleak/aws-lambda-functions-with-a-static-ip-89a3ada0b471) Gateway(attach lambda to VPC) or whitelist [all](https://ip-ranges.amazonaws.com/ip-ranges.json) your AWS Region Public IP/CIDR. – Imran Jul 10 '19 at 02:31
  • @Imran Maybe that is the problem, i did not whitelisted aws lambda on mongo db, do you have a tutorial to do that? – Rolly Jul 10 '19 at 18:05
  • correction to my earlier options. You have one more option provided by @Anton answer. You can perform AWS VPC peering between your AWS VPC(where lambda is attached) and MongoDB Atlas VPC. Check the articles provided in my earlier comment and Anton answer. – Imran Jul 10 '19 at 18:45

3 Answers3

1

Some extra steps are required to let lambda call external endpoint

https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/

Your atlas should also whitelist IP address of the servers, from which lambda will be connected.

Another option to consider - VPC peering between your lambda VPC and Atlas.

Anton
  • 3,587
  • 2
  • 12
  • 27
1

I have some questions concerning your configuration:

  1. Did you whitelist the AWS Lambda function's IP address in Atlas? Several posts on SO indicate that users get a MongoNetworkError like this if the IP is not whitelisted. [1][4]

  2. Did you read the best-practices guide by Atlas which states that mongodb connections should be initiated outside the lambda handler? [2][3]

  3. Do you use a public lambda function or a lambda function inside a VPC? There is a substantial difference between them and the latter one is more error-prone since the VPC configuration (e.g. NAT) must be taken into account.

I was able to ping the instances in the Atlas cluster and was able to establish a connection on port 27017. However, when connecting via the mongo shell, I get the following error:

Unable to reach primary for set CarpoolingCluster0-shard-0.

Cannot reach any nodes for set CarpoolingCluster0-shard-0. Please check network connectivity and the status of the set. This has happened for 1 checks in a row.

When I use your GitHub sample from AWS lambda I get the exact same error message as described in the question.

As the error messages are not authentication-related but network-related, I assume that something is blocking the connection... Please double-check the three config questions above.

[1] What is a TransientTransactionError in Mongoose (or MongoDB)?
[2] https://docs.atlas.mongodb.com/best-practices-connecting-to-aws-lambda/
[3] https://blog.cloudboost.io/i-wish-i-knew-how-to-use-mongodb-connection-in-aws-lambda-f91cd2694ae5
[4] https://github.com/Automattic/mongoose/issues/5237

Community
  • 1
  • 1
Martin Löper
  • 6,471
  • 1
  • 16
  • 40
  • I did not whitelisted the AWS ip in mongo atlas. I will search on google to see how do it. – Rolly Jul 10 '19 at 18:07
1

Well, thanks everyone. Finally i found the solution with the help of the mongo support. Here is the solution for anyone who needs

  1. When you create a Mongo Altas cluster they ask you add your local ip and it is automatically added to the WhiteList. You can see it in Your cluster > Network Access > IP Whitelist there in the list you will see your IP. It mean that only people from YOUR network will be able to connect to your MongoAtlas. AWS Lambda is NOT in your network, soo aws lambda will never connect to your Mongo Atlas. That is why i get the error MongoNetworkError.

Fix

  1. You need to add the AWS Lambda IP to the Mongo Atlas WhiteListIP
  2. go to your Your cluster > Network Access > IP Whitelist
  3. click in the button ADD IP ADDRESS
  4. click on ALLOW ACCESS FROM ANYWHERE it will add the ip 0.0.0.0/0 to the list, click confirm
  5. Test your call from AWS Lambda and i will work.

FINALLY !

What you did is tell to Mongo Atlas that ANYONE from ANYWHERE can connect to your mongo Atlas.

Of course this is not a good practice. What you need is add only the AWS Lambda IP, here is when VPC comes to scene. Create a VPC is little complex and it have many steeps, there are good tutorials in the other comments.

But for sure this small guide tacle the MongoNetworkError

Rolly
  • 3,205
  • 3
  • 26
  • 35