How to get npm to use cache

Question

UPDATE

I changed directions from this question and ended up taking advantage of Docker image layers to cache the npm install unless there is changes to the package.config, see here.

Note, in relation to this question, I still build my AngularJs Docker image in a slave Jenkins Docker image but I no longer run the npm install in the Docker slave, I copy my app files to my AngularJs Docker image and run the npm install in the AngularJs Docker image, thus getting a Docker cache layer of the npm install, inspiration from this great idea/answer here.

-------------------------------END UPDATE------------------------------

Ok, I should add the caveat that I am in a Docker container but that really shouldn't matter much possibly, I do not stop the container and I have volumes for the for the npm cache folder as well as the /home folder for the user running npm commands.

The purpose of the Docker container, with npm installed, is that it is a build slave, spun up by Jenkins to build an AngularJs application. The problem is that it is incredibly slow, downloading all the needed npm packages, every time.

jenkins is the user, a jenkins account on a build server is "whom" is running npm install

I have Volumes for both the npm folder for the user running the npm install cmd: /home/jenkins/.npm and also the folder that the command npm config get cache says is my cache directory: /root/.npm. Not that container volumes should even matter because I have not stopped the container after running npm install.

Ok the steps I take to start debugging, to start, I "bash into the container" with this command:

docker exec -it <container_id> bash

All commands I run from this point forward I am connected to the running container with npm installed.

echo "$HOME" results in /root

npm config get cache results in root/.npm

Any time jenkins runs npm install in this container, after that command finishes successfully, I run npm cache ls which always yields empty, nothing cached: ~/.npm

Many packages were downloaded however as we can see with ls -a /home/jenkins/.npm/:

So I tried setting the cache-min to a very long expiration time: npm config set cache-min 9999999 that didn't help.

I am not sure what else to do, it just seems that none of my npm packages are being cached, how do I get npm to cache packages?

here is a truncated npm install output:

Downloading binary from https://github.com/sass/node-sass/releases/download/v4.5.3/linux-x64-48_binding.node
Download complete
Binary saved to /home/jenkins/workspace/tsl.frontend.development/node_modules/node-sass/vendor/linux-x64-48/binding.node
Caching binary to /home/jenkins/.npm/node-sass/4.5.3/linux-x64-48_binding.node
Binary found at /home/jenkins/workspace/tsl.frontend.development/node_modules/node-sass/vendor/linux-x64-48/binding.node
Testing binary
Binary is fine
typings WARN deprecated 3/24/2017: "registry:dt/core-js#0.9.7+20161130133742" is deprecated (updated, replaced or removed)
[?25h
+-- app (global)
`-- core-js (global)

And here is my Dockerfile:

FROM centos:7
    MAINTAINER Brian Ogden

    RUN yum update -y && \
             yum clean all

    #############################################
    # Jenkins Slave setup
    #############################################
    RUN yum install -y \
            git \
            openssh-server \
            java-1.8.0-openjdk \
            sudo \
            make && \
            yum clean all

    # gen dummy keys, centos doesn't autogen them like ubuntu does
    RUN /usr/bin/ssh-keygen -A

    # Set SSH Configuration to allow remote logins without /proc write access
    RUN sed -ri 's/^session\s+required\s+pam_loginuid.so$/session optional pam_loginuid.so/' /etc/pam.d/sshd

    # Create Jenkins User
    RUN useradd jenkins -m -s /bin/bash

    # Add public key for Jenkins login
    RUN mkdir /home/jenkins/.ssh
    COPY /files/id_rsa.pub /home/jenkins/.ssh/authorized_keys

    #setup permissions for the new folders and files
    RUN chown -R jenkins /home/jenkins
    RUN chgrp -R jenkins /home/jenkins
    RUN chmod 600 /home/jenkins/.ssh/authorized_keys
    RUN chmod 700 /home/jenkins/.ssh

    # Add the jenkins user to sudoers
    RUN echo "jenkins  ALL=(ALL)  ALL" >> etc/sudoers
    #############################################

    # Expose SSH port and run SSHD
    EXPOSE 22
    #Technically, the Docker Plugin enforces this call when it starts containers by overriding the entry command. 
    #I place this here because I want this build slave to run locally as it would if it was started in the build farm.
    CMD ["/usr/sbin/sshd","-D"]

    #############################################
    # Docker and Docker Compose Install
    #############################################
    #install required packages
    RUN yum install -y \
        yum-utils \
        device-mapper-persistent-data \
        lvm2 \
        curl && \
        yum clean all

    #add Docker CE stable repository
    RUN yum-config-manager \
        --add-repo \
        https://download.docker.com/linux/centos/docker-ce.repo

    #Update the yum package index.
    RUN yum makecache fast

    #install Docker CE
    RUN yum install -y docker-ce-17.06.0.ce-1.el7.centos

    #install Docker Compose 1.14.0
    #download Docker Compose binary from github repo
    RUN curl -L https://github.com/docker/compose/releases/download/1.14.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
    #Apply executable permissions to the binary
    RUN chmod +x /usr/local/bin/docker-compose
    #############################################

    ENV NODE_VERSION 6.11.1

    #############################################
    # NodeJs Install
    #############################################
    RUN yum install -y \
            wget

    #Download NodeJs package
    RUN wget https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.gz

    #extract the binary package into our system's local package hierarchy with the tar command. 
    #The archive is packaged within a versioned directory, which we can get rid of by passing the --strip-components 1 option. 
    #We will specify the target directory of our command with the -C command:
    #This will install all of the components within the /usr/local branch
    RUN tar --strip-components 1 -xzvf node-v* -C /usr/local
    #############################################

#############################################
# npm -setup volume for package cache
# this will speed up builds
#############################################
RUN mkdir /home/jenkins/.npm
RUN chown jenkins /home/jenkins/.npm .
RUN mkdir /root/.npm
RUN chown jenkins /root/.npm .
#for npm cache, this cannot be expressed in docker-compose.yml
#the reason for this is that Jenkins spins up slave containers using
#the docker plugin, this means that there
VOLUME /home/jenkins/.npm
VOLUME /root/.npm
#############################################

I am confused... by "in a Docker container" do you mean you are executing this using `docker run` or `docker exec`? — Derek Brown, Oct 25 '17 at 01:01
I just docker exec -it bash into the container to run the cmds — Brian Ogden, Oct 25 '17 at 01:02
Can I ask why? This isn't really the intended functionality for Docker... — Derek Brown, Oct 25 '17 at 01:11
@DerekBrown caching npm packages isn't intended for Docker? why do you say that? — Brian Ogden, Oct 25 '17 at 01:15
I mean that using bash within Docker isn't really the intended functionality. Why not just build a Dockerfile which builds your package, and use Docker's build cache? — Derek Brown, Oct 25 '17 at 01:16
what happens when you run `ls -a /home/jenkins/.npm/` after install? — Derek Brown, Oct 25 '17 at 01:17
What are you talking about, I am doing testing here I am not regular bashing into the container to run npm install, I am debugging right now — Brian Ogden, Oct 25 '17 at 01:18
@DerekBrown there are many packages when listing files in ls -a /home/jenkins/.npm/ — Brian Ogden, Oct 25 '17 at 01:18
@DerekBrown there are many packages, that is where npm is downloading the packages is too it looks like — Brian Ogden, Oct 25 '17 at 01:19
@DerekBrown I added a screenshot of ls -a /home/jenkins/.npm/ — Brian Ogden, Oct 25 '17 at 01:22
@DerekBrown this Docker container is a build slave, spun up by Jenkins to build an AngularJs application — Brian Ogden, Oct 25 '17 at 01:22
@DerekBrown yes, the 5 minute builds have become inconvenient, there are times that we are waiting on the build to complete into a meeting where we could instead be looking at the frontend changes, when I build localhost, once packages are cached it takes 20 seconds — Brian Ogden, Oct 25 '17 at 01:32
try `su jenkins` and then do the `npm install` and see if it runs from cache. — Derek Brown, Oct 25 '17 at 01:36
@DerekBrown it does not seem to be running from the cache, su jenkins and then npm install seems to be downloading lots of packages for the first time, still running — Brian Ogden, Oct 25 '17 at 01:39
wait for that to finish, then try `npm install` again. I imagine this will make your cache persistent — Derek Brown, Oct 25 '17 at 01:41
@DerekBrown you are correct, for the first time npm cache ls is yielding results — Brian Ogden, Oct 25 '17 at 01:42
@DerekBrown after su jenkins and npm install, when running npm config get cache the result is /home/jenkins/.npm — Brian Ogden, Oct 25 '17 at 01:44

score 6 · Accepted Answer · answered Oct 25 '17 at 01:45

6

When you run docker exec -it <container> bash you access the Docker container as the root user. npm install thus saves the cache to /root/.npm, which isn't a volume saved by the container. Jenkins, on the other hand, uses the jenkins user, which saves to /home/jenkins/.npm, which is being cached. So in order to emulate the functionality of the actual Jenkins workflow, you need to su jenkins before you can npm install.

That being said, the npm cache is not a perfect solution (especially if you have a ton of automated Jenkins builds). Some things to look into that would be better long-term solutions:

Install a local NPM Cache like sinopia. I found this guide to be particularly helpful.
Use Docker to build you app (which would work fine with Docker In Docker). Docker would cache after each build step, saving the repeated fetching of dependencies.

answered Oct 25 '17 at 01:45

Derek Brown

4,232
4
27
44

thanks, so we have established that the cache is indeed working, at least according to npm cache ls. Though I must admit, npm still doesn't seem to be using the cache for anything, an http call is made for every package – Brian Ogden Oct 25 '17 at 01:59
@BrianOgden Have you tried combining this approach with the `cache min` one you tried earlier? – Derek Brown Oct 25 '17 at 02:01
Ya know the volume I am using for the Docker container is dangling, getting orphaned so that is my next problem to deal with – Brian Ogden Oct 25 '17 at 04:05

How to get npm to use cache

1 Answers1