56

We have a fairly simple node.js app, but due to AWS Elastic Beanstalk deployment mechanism, it takes about 5 minutes to roll-out a new version (via git aws.push) even after a single file commit.

I.e. the commit itself (and upload) is fast (only 1 file to push), but then Elastic Beanstalk fetches whole package from S3, unzips it and runs npm install, which causes node-gyp to compile some modules. Upon installation/building completion, Elastic Beanstalk wipes /var/app/current and replaces it with the new app version.

Needless to say, constant node_modules rebuilding is not necessary, and rebuilding that takes 30 seconds on my old Macbook Air, takes >5 mins on a ec2.micro instance, not fun.

I see two approaches here:

  1. tweak /opt/containerfiles/ebnode.py and play with node_modules location to avoid its removal and rebuilding upon deployment.
  2. set up a git repo on Elastic Beanstalk EC2 instance and basically re-write deployment procedure ourselves, so /var/app/current receives pushes and runs npm install only when necessary (which makes Elastic Beanstalk to look like OpsWorks..)

Both options lack grace and are prone to issues when Amazon updates their Elastic Beanstalk hooks and architecture.

Maybe somebody has a better idea how to avoid constant rebuilding of node_modules that are already present in the app dir? Thank you.

Ryan Parman
  • 6,855
  • 1
  • 29
  • 43
Kirill Kay
  • 787
  • 1
  • 7
  • 16
  • Unfortunately I don't have a better idea. I struggled with this, along with a list of other issues (no support for recent Node versions, non-deterministic deployment results), and in the end I went the self-managed EC2 route instead. – Jorge Aranda Jan 18 '14 at 05:28
  • Thanks for your input. Are there any best practices on how to automate node.js deployment to EC2 via git hooks and such? What about automated scaling, monitoring and all that jazz that ELB boasts about? – Kirill Kay Jan 18 '14 at 05:36
  • As a note, Amazon Elastic Load Balancing (ELB) is different from AWS Elastic Beanstalk. I had to re-read this post a couple of times wondering why you were messing with the load balancer. :) – Ryan Parman Jan 18 '14 at 08:30
  • My bad, it's easy to get lost in AWS abbreviations :) – Kirill Kay Jan 18 '14 at 09:42

5 Answers5

41

Thanks Kirill, it was really helpful !

I'm just sharing my config file for people who just look the simple solution to the npm install. This file needs to be placed in the .ebextensions folder of the project, it is lighter since it doesn't include last version of node installation, and ready to use.

It also dynamically checks the node version installed, so no need for it to be included in the env.vars file.

.ebextensions/00_deploy_npm.config

files:
  "/opt/elasticbeanstalk/env.vars" :
    mode: "000775"
    owner: root
    group: users
    content: |
      export NPM_CONFIG_LOGLEVEL=error
      export NODE_PATH=`ls -td /opt/elasticbeanstalk/node-install/node-* | head -1`/bin
  "/opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh" :
    mode: "000775"
    owner: root
    group: users
    content: |
      #!/bin/bash
      . /opt/elasticbeanstalk/env.vars
      function error_exit
      {
        eventHelper.py --msg "$1" --severity ERROR
        exit $2
      }

      #install not-installed yet app node_modules
      if [ ! -d "/var/node_modules" ]; then
        mkdir /var/node_modules ;
      fi
      if [ -d /tmp/deployment/application ]; then
        ln -s /var/node_modules /tmp/deployment/application/
      fi

      OUT=$([ -d "/tmp/deployment/application" ] && cd /tmp/deployment/application && $NODE_PATH/npm install 2>&1) || error_exit "Failed to run npm install.  $OUT" $?
      echo $OUT
  "/opt/elasticbeanstalk/hooks/configdeploy/pre/50npm.sh" :
    mode: "000666"
    owner: root
    group: users
    content: |
       #no need to run npm install during configdeploy
Tronix117
  • 1,985
  • 14
  • 19
  • 2
    thanks for your comment. Yes, I guess most people looking only to speed up npm rebuild times will benefit from your config. I'll make it a default answer, but in case someone would like a more sophisticated solution, please refer to my answer or better-faster-elastic-beanstalk repo: http://github.com/kopurando/better-faster-elastic-beanstalk – Kirill Kay Apr 23 '14 at 16:04
  • 1
    @Tronix117 Running this straight out of the box gives us `ERROR: Failed to run npm install. /usr/bin/env: node: No such file or directory` Any idea what might be causing this? – Mirage Oct 11 '16 at 13:42
  • 1
    @Mirage It seems that npm can not be launched, because node is not a know command on the system, seems to be a PATH issue. You can resolve it by changing `$NODE_PATH/npm install` to `$NODE_PATH/node $NODE_PATH/npm install` to force using the `node` binary within the `$NODE_PATH` to launch `npm` – Tronix117 Oct 11 '16 at 21:11
  • I tried that, it gave a whole other set of permission denied errors (couldn't write to logs, did not have access to the NPM_TOKEN env var) and so on and so forth. Decided to skip the caching for now. – Mirage Oct 12 '16 at 10:04
  • @mirage yes, you should have something wrong with your config. But actualy this script has been made 2 years ago, since then, AWS changed a few stuff. The idea is here, but it can need some tweaks for it to work nowadays. – Tronix117 Oct 12 '16 at 12:49
  • 3
    For me, this failed with error `/usr/bin/env: node: No such file or directory`, I fixed it by adding `PATH="$PATH:$NODE_PATH"` on line 29, one line above `OUT=$(...)`. – user12341234 May 14 '17 at 17:55
38

25/01/13 NOTE: updated scripts to run npm -g version upgrade (only once, on initial instance roll out or rebuild) and to avoid NPM operations during EB configuration change (when app dir is not present, to avoid error and to speed up configuration updates).

Okay, Elastic Beanstalk behaves dodgy with recent node.js builds (including presumably supported v.0.10.10), so I decided to go ahead and tweak EB to do the following:

  1. to install ANY node.js version as per your env.config (including the most recent ones that are not yet supported by AWS EB)
  2. to avoid rebuilding existing node modules, including in-app node_modules dir
  3. to install node.js globally (and any desired module as well).

Basically, I use env.config to replace deploy&config hooks with customized ones (see below). Also, in a default EB container setup some env variables are missing ($HOME for example) and node-gyp sometimes fails during rebuild because of it (took me 2 hours of googling and reinstalling libxmljs to resolve this).

Below are the files to be included along with your build. You can inject them via env.config as inline code or via source: URL (as in this example)

env.vars (desired node version & arch are included here and in env.config, see below)

export HOME=/root
export NPM_CONFIG_LOGLEVEL=error
export NODE_VER=0.10.24
export ARCH=x86
export PATH="$PATH:/opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/:/root/.npm"

40install_node.sh (fetch and ungzip desired node.js version, make global symlinks, update global npm version)

#!/bin/bash
#source env variables including node version
. /opt/elasticbeanstalk/env.vars

function error_exit
{
  eventHelper.py --msg "$1" --severity ERROR
  exit $2
}

#UNCOMMENT to update npm, otherwise will be updated on instance init or rebuild
#rm -f /opt/elasticbeanstalk/node-install/npm_updated

#download and extract desired node.js version
OUT=$( [ ! -d "/opt/elasticbeanstalk/node-install" ] && mkdir /opt/elasticbeanstalk/node-install ; cd /opt/elasticbeanstalk/node-install/ && wget -nc http://nodejs.org/dist/v$NODE_VER/node-v$NODE_VER-linux-$ARCH.tar.gz && tar --skip-old-files -xzpf node-v$NODE_VER-linux-$ARCH.tar.gz) || error_exit "Failed to UPDATE node version. $OUT" $?.
echo $OUT

#make sure node binaries can be found globally
if [ ! -L /usr/bin/node ]; then
  ln -s /opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/node /usr/bin/node
fi

if [ ! -L /usr/bin/npm ]; then
ln -s /opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/npm /usr/bin/npm
fi

if [ ! -f "/opt/elasticbeanstalk/node-install/npm_updated" ]; then
/opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/ && /opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/npm update npm -g
touch /opt/elasticbeanstalk/node-install/npm_updated
echo "YAY! Updated global NPM version to `npm -v`"
else
  echo "Skipping NPM -g version update. To update, please uncomment 40install_node.sh:12"
fi

50npm.sh (creates /var/node_modules, symlinks it to app dir and runs npm install. You can install any module globally from here, they will land in /root/.npm)

#!/bin/bash
. /opt/elasticbeanstalk/env.vars
function error_exit
{
  eventHelper.py --msg "$1" --severity ERROR
  exit $2
}

#install not-installed yet app node_modules
if [ ! -d "/var/node_modules" ]; then
  mkdir /var/node_modules ;
fi
if [ -d /tmp/deployment/application ]; then
  ln -s /var/node_modules /tmp/deployment/application/
fi

OUT=$([ -d "/tmp/deployment/application" ] && cd /tmp/deployment/application && /opt/elasticbeanstalk/node-install/node-v$NODE_VER-linux-$ARCH/bin/npm install 2>&1) || error_exit "Failed to run npm install.  $OUT" $?
echo $OUT

env.config (note node version here too, and to be safe, put desired node version in env config in AWS console as well. I'm not certain which of these settings will take precedence.)

packages:
  yum:
    git: []
    gcc: []
    make: []
    openssl-devel: []

option_settings:
  - option_name: NODE_ENV
    value: production
  - option_name: RDS_HOSTNAME
    value: fill_me_in
  - option_name: RDS_PASSWORD
    value: fill_me_in
  - option_name: RDS_USERNAME
    value: fill_me_in
  - namespace: aws:elasticbeanstalk:container:nodejs
    option_name: NodeVersion
    value: 0.10.24

files:
  "/opt/elasticbeanstalk/env.vars" :
    mode: "000775"
    owner: root
    group: users
    source: https://dl.dropbox.com/....
  "/opt/elasticbeanstalk/hooks/configdeploy/pre/40install_node.sh" :
    mode: "000775"
    owner: root
    group: users
    source: https://raw.github.com/....
  "/opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh" :
    mode: "000775"
    owner: root
    group: users
    source: https://raw.github.com/....
  "/opt/elasticbeanstalk/hooks/configdeploy/pre/50npm.sh" :
    mode: "000666"
    owner: root
    group: users
    content: |
       #no need to run npm install during configdeploy
  "/opt/elasticbeanstalk/hooks/appdeploy/pre/40install_node.sh" :
    mode: "000775"
    owner: root
    group: users
    source: https://raw.github.com/....

There you have it: on t1.micro instance deployment now takes 20-30 secs instead of 10-15 minutes! If you deploy 10 times a day, this tweak will save you 3 (three) weeks in a year. Hope it helps and special thanks to AWS EB staff for my lost weekend :)

Kirill Kay
  • 787
  • 1
  • 7
  • 16
  • Thank you for taking the time to write a detailed follow-up to your own question, incredibly helpful! – Elliot Chong Mar 02 '14 at 20:17
  • Thanks for the great writeup. Quick clarification ... when you say, "to avoid rebuilding existing node modules, including in-app node_modules dir", are you saying that you added your application node_modules to source control? Thanks! – mveerman Mar 12 '14 at 14:15
  • 2
    I meant, these hooks will avoid unnecessary rebuilding of binaries (i.e. running node-gyp) within NPM modules, installed globally and locally (inside app dir). Updated modules versions still will be installed and rebuilt (if necessary), but if nothing has changed in package.json since last deployment, no extra actions will be taken by NPM and deploy will take less than a minute (on t1.micro instance). – Kirill Kay Mar 13 '14 at 15:27
  • Thanks, for some reason I missed your symlink, ln -s /var/node_modules /tmp/deployment/application/, thanks! – mveerman Mar 13 '14 at 18:01
  • This is amazing! Just wondering how this will work with the autoscaling? – Steeve17 Mar 29 '14 at 18:24
  • 2
    I have created a [gist](https://gist.github.com/etiennea/9861792) so we can improve this. I have added --production to npm install since no need to install testing frameworks on the server. Also added export NPM_CONFIG_PRODUCTION=true which does the same. Did not know which method was better – Steeve17 Mar 29 '14 at 20:00
  • 2
    nice addition! I kinda open-sourced my hooks, as they're getting more sophisticated and I needed public repo to keep the up-to-date files available for any elastic beanstalk instance. Feel free to fork or participate: https://github.com/kopurando/better-faster-elastic-beanstalk Forking my hooks for your own purposes is a good idea as I keep adding stuff that is probably needed only for my own project. – Kirill Kay Mar 31 '14 at 05:26
  • recent additions: - update nginx binaries (i.e. custom built nginx of any version/modules configuration) - added PhantomJS - tweak nginx configs - improved logging (it's more realtime now, so log.io can be a good measure of deployment process) – Kirill Kay Mar 31 '14 at 05:28
  • as for scaling, I'm about to test it this week, will keep this topic and BFEB updated. – Kirill Kay Mar 31 '14 at 06:02
  • Hi Kirill. Thanks for these amazing notes! How did the auto scaling work? – Ted Benson May 18 '14 at 04:40
  • We haven't had a chance to test in autoscaling environment yet, as it turned out that larger instances spread across regions currently suit our use-case better. Personally, I can't envision any problems with configs described, especially if you consider using leader_only option to avoid command clashes on multiple instances, for example: ` 04_setup_crontab:` ` command: crontab /tmp/cronjob && rm /tmp/cronjob` ` leader_only: true` Moreover, I'm pretty sure you can check on the instance whether it's a leader or not and conditionally run (or don't run) specific hooks or scripts. – Kirill Kay Jun 06 '14 at 17:31
  • I believe symlinking doesn't works anymore in node.js v6. Removing them I could `npm install` okay. – André Werlang May 18 '16 at 17:33
  • Also, current EB requires a supported NodeVersion. – André Werlang May 18 '16 at 17:42
  • if you have time, would you mind creating a pull request in https://github.com/kopurando/better-faster-elastic-beanstalk? As I have been out of development for more than a year now, and do not follow latest EB trends.. – Kirill Kay May 21 '16 at 03:27
5

There's npm package that's overwriting default EB behaviour for npm install command by truncating following files:

  • /opt/elasticbeanstalk/hooks/appdeploy/pre/50npm.sh
  • /opt/elasticbeanstalk/hooks/configdeploy/pre/50npm.sh

https://www.npmjs.com/package/eb-disable-npm

Might be better than just copying script from SO, since this package is maintained and probably will be updated when EB behaviour will change.

panK
  • 111
  • 2
  • 5
2

I've found a quick solution to this. I looked through the build scripts that Amazon are using and they only run npm install if package.json is present. So after your initial deploy you can change it to _package.json and npm install won't run anymore! It's not the best solution but it's a quick fix if you need one!

cchapman
  • 3,269
  • 10
  • 50
  • 68
Peter
  • 104
  • 3
  • 1
    but how will you get modules installed or updated w/o package.json? – Kirill Kay Dec 18 '15 at 07:10
  • I'm just using it for a small project at the moment with a small number of dependencies, so I'm uploading the `node_modules`. However if you're not adding node modules often you could leave `package.json` there for the initial deploy and then change it. That way they're installed but rebuild is never run. Otherwise this quick (and hopefully temporary) solution isn't for you! – Peter Dec 21 '15 at 09:58
-7

I had 10+ minute builds when I would deploy. The solution was much simpler than others have came up with... Just check node_modules into git! See http://www.futurealoof.com/posts/nodemodules-in-git.html for the reasoning

Eric
  • 586
  • 1
  • 5
  • 19
  • 1
    Elastic Beanstalk seems to be calling rebuild with NPM. I don't know if this is a recent change, but for compiled stuff this wouldn't really help as far as I can tell. – Brad Jul 23 '14 at 15:02
  • 2
    This won't work for compiled libraries - example: zmq. If you build on mac, it won't deploy to linux. Just something to keep in mind. – Brad Gunn Feb 27 '15 at 16:35
  • 1
    My `node_modules` for a basic webpack+react+koa app is 500MB. Pass. However, the pain I have felt from remote dep management might justify it. – vaughan May 20 '15 at 17:13
  • The link has died. This seems to work well for us. Our solution in detail was to 1) include modules we don't want to be run through "npm install" in .gitignore, 2) move those modules in packages.json to devDependencies section, 3) run npm install without --production flag on the deployment machine, 4) deploy to EB – h-kippo Sep 29 '15 at 10:25