19

I'm using AWS Lambda, which involves creating an archive of my node.js script, including the node_modules folder and uploading that to their infrastructure to run.

This works fine, except when it comes to node modules with native bindings (using node-gyp). Because the binding was complied and project archived on my local computer (OS X), it is not compatible with AWS's (Amazon Linux) servers.

How can I cross-compile/install a node module (specifically, node-sqlite3) so when I upload it to another server arch it runs?

Josh Hunt
  • 14,225
  • 26
  • 79
  • 98
  • Have you tried utilizing node-pre-gyp? node-sqlit3 already uses it, so you're ahead of the game there. IIRC, it should offload the build to Amazon S3, so you can compile against Amazon Linux (AWS Lambda's OS). – zamnuts May 23 '15 at 01:57
  • Yeah, I saw that node-sqlite3 uses node-pre-gyp, but I didn't really understand what that means for me or how to use it to achieve what I want. – Josh Hunt May 23 '15 at 01:58
  • I know it involves additionnal cost, but i d suggest to package ad build onto another s3 box. Then transfer it to your production environment. Note that if you d do npm i from your linux environment, everything would work fine out of the box if you have the required build tools. –  May 23 '15 at 02:46
  • Does this help any? http://stackoverflow.com/questions/25797424/cross-compile-node-js-contextify-for-arm It basically says you need to have the right environment variables for it to know which compiler to use. – Randy May 23 '15 at 18:26
  • Yeah that's a start, but I have no idea what compilers I need or what to set the env vars to. I've tried looking up and finding what I need, but found it overly difficult to do so. – Josh Hunt May 25 '15 at 12:33

4 Answers4

9

While not really a solution to your problem, a very easy workaround could be to simply compile the native addons on a Linux machine.

For your particular situation, I would use Vagrant. Vagrant can create virtual machines and configure them within seconds.

  1. Find an OS image that resembles Amazon's Linux distro (Fedora, CentOS, others that use yum as package manager - see Wiki)
  2. Use a simple configuration script that, when run by Vagrant on machine startup, will run npm install (optionally it might also remove the node_modules folder before to ensure a clean installation)
  3. For extra comfort, the script can also create the zip file for deployment
  4. Once the installation finishes, the script will shutdown the VM to avoid unnecessary consumption of system resources
  5. Deploy!

It might require some tuning if the linked libraries are not at the same place on the target machine but generally this seems to me like the best and quickest solution.

Robert Rossmann
  • 11,931
  • 4
  • 42
  • 73
2

While installing the app using Vagrant might be sufficient in some cases, I have found it necessary to build the app on Linux which is as close to Lambda's Amazon Linux AMI as possible.

You can read the original answer here: https://stackoverflow.com/a/34019739/303184

Steps to make it work:

  1. Spawn new EC2 instance. Make sure it is based on exactly the same image as your AWS Lambda runtime. You can review Lambda env details here: http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html. In our case, it was Amazon Linux AMI called amzn-ami-hvm-2015.03.0.x86_64-gp2.

  2. Install nvm and use it to install the same version of Node.js as on the AWS Lambda. At the time of writing this, it was v0.10.36. You can refer to http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html again to find out.

  3. You will probably need to install git & g++ compiler on the EC2. You can do this running

    sudo yum install git gcc-c++

  4. Finally, clone your app to your new EC2 and install your app's dependecies:

    nvm use 0.10.36
npm install --production

  5. You can then easily download the node_modules using scp or such.

Community
  • 1
  • 1
xaralis
  • 4,634
  • 1
  • 23
  • 20
  • While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - [From Review](/review/low-quality-posts/10399857) – Eloims Dec 01 '15 at 12:39
  • @Eloims I have attached the full solution as requested. – xaralis Dec 02 '15 at 13:21
1

Same lines as Robert's answer, when I had to work on my MAC in a different OS I use vm ware like Oracle's free virtualizer VirtualBox to get a linux on my mac, no cost to me. Or sign up for a new AWS account, you get a micro for a year free. Use that to get your linux box, do whatever you need there.

tgkprog
  • 4,493
  • 4
  • 41
  • 70
1

AWS has a page describing how to deal with native NPM modules: https://aws.amazon.com/blogs/compute/nodejs-packages-in-lambda/

bspates
  • 382
  • 1
  • 10