5

We have Jenkins running within ECS. We are using pipelines for our build and deploy process. The pipeline uses the docker plugin to pull an image which has some dependencies for testing etc, all our steps then occur within this docker container.

The issue we currently have is that our NPM install takes about 8 minutes. We would like to speed this process up. As containers are being torn down at the end of each build then the node_modules that are generated are disposed of. I've considered NPM caching but due to the nature of docker this seemed irrelevant unless we pre-install the dependencies into the docker image (but this triples the size of the image almost). Are there simple solutions to this that will help our NPM install speeds?

RB26
  • 51
  • 1
  • 1
  • 3

4 Answers4

9

You should be using package caching but not caching node_modules directly. Instead you mount cache directories that your package installer uses, and your install will be blazing fast. Docker does make that possible by allowing you to mount directories in a container, that persist across builds.

For yarn mount ~/.cache or ~/.cache/yarn
For npm mount ~/.npm

docker run -it -v ~/.npm:/.npm ~/.cache:/.cache /my-app:/my-app testing-image:1.0.0 bash -c 'npm ci && npm test`

Note: I'm using npm ci here, which will always delete node_modules and reinstall using exact versions in the package-lock.json, so you get very consistent builds. (In yarn, this is yarn install --frozen-lockfile)

kuceb
  • 16,573
  • 7
  • 42
  • 56
1

You could set up a Http proxy and cache all dependencies (*)(**).

Then use --build-arg to set HTTP_PROXY variable:

docker build --build-arg HTTP_PROXY=http://<cache ip>:3128 .

*: This will not work improve performance on dependencies that need to be compiled (ie: c/c++ bindings)

**: I use a Squid container to share cache configuration

Gonzalo Matheu
  • 8,984
  • 5
  • 35
  • 58
0

In my case it was a bunch of corporate software installed in my computer apparently some anti virus analyzing all the node_modules files from the container when I mounted the project folder on the host machine, what I did was avoid mounting node_modules locally. Immediately sped up from 25 min to 5.

Rafalfaro
  • 211
  • 4
  • 3
0

I have explained what I did with a possible implementation here. I have not used the package-lock.json but the npm ls command to check for changes in the node_modules folder so that I could potentially skip the step of re-uploading the cached modules on the bind mount.

@bkucera 's answer points you in the right direction with the bind mount, in general the easiest option in a containerized environment is to create a volume storing the cached packages. These packages could be archived in a tarball, which is the most common option, or even compressed if necessary (files in a .tar are not compressed).

Marco Luzzara
  • 5,540
  • 3
  • 16
  • 42