How can I set up MongoDB on a Node.js server using node-mongodb-native in an EC2 environment?

Question

I got help from many people here, and now I want to contribute back. For those who are having trouble making a Node.js server work with MongoDB, here is what I've done.

By the way, it is just the tip of the iceberg. I still have to explore 1) how to make replicate mongoDB, 2) how to make replicate mongoDB locally, 3) how to do sharding, 4) multiple instance and load balancing, 5) how to manage multiple instances with nodejs server, 6) how to manage local data in each instance. There are lots of things I have to look up. :) If you have any reference, please let me know. — murvinlai, Jan 25 '11 at 01:19
for load balancing use HAProxy: http://stackoverflow.com/questions/4360221/haproxy-websocket-disconnection/4737648#4737648 — Shripad Krishna, Jan 25 '11 at 03:08
mongodb needs to run on its own instance if you planning to load balance your node app using HAProxy. It makes it easy to shard your db. — Shripad Krishna, Jan 25 '11 at 03:09
ok. I will take a look. Do you know if I use AWS load balancing service, will it work? also, I still don't understand one thing. If each instance save data locally, then that's mean if I have 10 instances, then it will save in 10 places? and each instance may have different data? — murvinlai, Jan 25 '11 at 11:41
Please put this in the form of a question, then move the answer part down to an answer. I appreciate what you're trying to do, but please keep it in the form of Q&A. Thanks. — Bill the Lizard, Jan 25 '11 at 13:28

score 10 · Accepted Answer · answered Jan 28 '11 at 17:55

This was originally posted by the question asker. A mod asked him in the comments to post it as an answer, but got no response. So, I cleaned it up and am posting it myself.

When you look at the code, you will notice that the createServer code is inside db.open. It won't work if you reverse it. Also, do not close the db connection. Otherwise, after the first time, the db connection will not be opened again. (Of course, db.open is declared outside of createServer.) I have no clue why createServer is inside db.open. I guess it may have to do with not opening too many db connections?

Also, one problem I face is that when I run it via SSH, even if I run the server in the background (e.g. $ node server.js &), after 2.5 hours, the server dies (not the instance though). I am not sure if it is because of terminal connection or what.

Here is the procedure & code

Environment: EC2, AMS-Linux-AMI

Purpose: Take an HTTP request and log the query, IP and timestamp into MongoDB.

Steps

1) After creating the instance (server), install gcc.

$ yum install gcc-c++

2) Download Node.js files and unzip them. (I used version 2.6.)

$ curl -O http://nodejs.org/dist/node-v0.2.6.tar.gz
$ tar -xzf node-v0.2.6.tar.gz

I renamed the unzipped folder to just "nodejs"

$ cd nodejs
$ sudo ./configure --without-ssl
$ sudo make
$ sudo make install

make takes a long while.... After that you can try running the sample in nodejs.org

3) Install MongoDB. I installed version 1.6.5, not 1.7.

$ curl -O http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-1.6.5.tgz
$ tar -xzf mongodb-linux-x86_64-1.6.5.tgz
$ sudo mkdir /data/db/r01/

I renamed the folder to "mongodb"

Run the db process:

$ ./mongodb/bin/mongod --dbpath /data/db/r01/

Then if you like, you can run and try out the command line. Refer to MongoDB's website.

4) I recommend that you create your own AIM based on your instance. It will take 20 minutes. Then, recreate the install and run MongoDB again.

5) Install node-mongodb-native

$ curl -O https://download.github.com/christkv-node-mongodb-native-V0.8.1-91-g54525d8.tar.gz
$ tar -xzf christkv-node-mongodb-native-V0.8.1-91-g54525d8.tar.gz

I renamed the folder to node-mongodb-native

$ cd node-mongodb-native
$ make

6) Here is the code for the server:

GLOBAL.DEBUG = true;
global.inData = '';
var http = require('http');
sys = require("sys");

/* set up DB */

var Db = require('./node-mongodb-native/lib/mongodb').Db,
  Connection = require('./node-mongodb-native/lib/mongodb').Connection,
  Server = require('./node-mongodb-native/lib/mongodb').Server,
  BSON = require('./node-mongodb-native/lib/mongodb').BSONNative;

var host = process.env['MONGO_NODE_DRIVER_HOST'] != null ? process.env['MONGO_NODE_DRIVER_HOST'] : 'localhost';
var port = process.env['MONGO_NODE_DRIVER_PORT'] != null ? process.env['MONGO_NODE_DRIVER_PORT'] : Connection.DEFAULT_PORT;
var db = new Db('test01', new Server(host, port, {}), {native_parser:true});

db.open(function(err, db) { 
    http.createServer(function (req, res) {
        res.writeHead(200, {'Content-Type': 'text/plain'});

        global.inData = {'p':'', 'url':''};

        // get IP address
        var ipAddress = req.connection.remoteAddress;
        global.inData.ip = ipAddress;

        // date time
        var d = new Date();
        var ts = d.valueOf();
        global.inData.ts = ts;

        // get the http query
        var qs = {};
        qs = require('url').parse(req.url, true);
        if (qs.query !== null) {
            for (var key in qs.query) {
                if (key == 'p') {
                    global.inData.p = qs.query[key];
                }
                if (key == 'url') {
                    global.inData.url = qs.query[key];
                }
            }
        }

        if (global.inData.p == '' && global.inData.url == '') {
            res.end("");
        } else {
            db.collection('clickCount', function(err, collection) { 
                if (err) {
                    console.log('is error \n' + err);
                }

                collection.insert({'p':global.inData.p, 
                  'url':global.inData.url,
                  'ip':global.inData.ip, 
                  'ts':global.inData.ts});
                res.end("");
                //db.close();  // DO NOT CLOSE THE CONNECTION
            }); 
        }
    }).listen(8080); 
});

console.log('Server running at whatever host :8080');

This may not be perfect code, but it runs. I'm still not used to the "nested" or LISP kind of coding style. That's why I cheated and used global.inData to pass data along. :)

Don't forget to put res.end("") in the appropriate location (where you think the HTTP request call should be ended).

thanks.. I wasn't no response though but just misunderstand what the mod wants me to do. :) anyway, thanks for your help. — murvinlai, Feb 08 '11 at 00:32

murvinlai · Answer 2 · 2011-05-10T17:13:56.843

By the way, the answer I posted above works for CentOS and Fedora.

For people who have Ubuntu, here it is:

# for Gcc
$ sudo apt-get install build-essential

# for SSL
$ sudo apt-get install libssl-dev

Then just install node.js and mongodb as described above.

Also, after few months of development, I find out using "npm", "express" and "mongoose" can bring my life much easier. Also, I have installed other tools, like debugger.

# Install Node Package Manager
$ sudo curl http://npmjs.org/install.sh | sh

# for debugging
$ sudo npm install node-inspector

# for Profiling
$ sudo npm install profile

# Install Express, the Node.js framework
$ sudo npm install express

# Install Template Engines (Now, let’s install Jade, jQuery Templates and EJS. You can pick the one you want)
$ sudo npm install jade jqtpl ejs

# XML related, install node-expat and then node-xml2js-expat
$ sudo apt-get install -y libexpat1-dev
$ sudo npm install node-xml2js
$ sudo npm install xml2js-expat

# Install Mongoose, (Mongo Driver)
$ sudo npm install mongoose

Reference: http://npmjs.org

http://expressjs.com

http://mongoosejs.com

score 0 · Answer 3 · answered Jan 08 '12 at 20:32

0

It looks like there might be a bug. It won't allow me to use a var for the first argument in a:b inside collection.insert({

It is treating first agument as 'a' or a, hard coded, either way.

I will look into this and post a fix on github

answered Jan 08 '12 at 20:32

I posted it a long while ago. Now I'm using Expressjs and Mongoose (and Mongolian). I suggest you to use the same thing and make your life easier. :D – murvinlai Jan 09 '12 at 11:01

How can I set up MongoDB on a Node.js server using node-mongodb-native in an EC2 environment?

3 Answers3

Linked