gitlab 8.2.1, How to use cache in .gitlab-ci.yml

Question

I'm trying to use 'cache' in .gitlab-ci.yml (http://doc.gitlab.com/ce/ci/yaml/README.html#cache). My gitlab version is 8.2.1 and my Runner is:

$ docker exec -it gitlab-runner gitlab-runner -v 
gitlab-runner version 0.7.2 (998cf5d)

So according to the doc, everything is up to date, but I'm unable to use the cache ;-(. All my files are always deleted. Am I doing something wrong?

A cache archive is created, but not passed to the next jobs.

Here is my .gitlab-ci.yml

$ cat .gitlab-ci.yml
    stages:
    - createcache
    - testcache

    createcache:
      type: createcache
      cache:
        untracked: true
        paths:
          - doc/
      script:
        - touch doc/cache.txt

    testcache:
      type: testcache
      cache:
        untracked: true
        paths:
          - doc/
      script:
        - find .
        - ls doc/cache.txt

Output of the job 'createcache'

Running on runner-141d90d4-project-2-concurrent-0 via 849d416b5994...
Fetching changes...
HEAD is now at 2ffbadb MUST BE REVERTED
[...]
$ touch doc/cache.txt
[...]
Archiving cache...
INFO[0000] Creating archive cache.tgz ...              
INFO[0000] Done!                                        

Build succeeded.

Output of the job 'testcache'

Running on runner-141d90d4-project-2-concurrent-0 via 849d416b5994...
Fetching changes...
Removing doc/cache.txt
[...]
$ ls doc/cache.txt
ls: cannot access doc/cache.txt: No such file or directory

ERROR: Build failed with: exit code 1

My workaround

My workaround is to manually untar what's in the /cache directory ... I'm pretty sure that's not the correct way to use cache ...

$ cat .gitlab-ci.yml
    stages:
    - build
    - test
    - deploy

    image: ubuntu:latest

    before_script:
      - export CACHE_FILE=`echo ${CI_PROJECT_DIR}/createcache/${CI_BUILD_REF_NAME}/cache.tgz | sed -e "s|/builds|/cache|"`

    createcache:
      type: build
      cache:
        untracked: true
        paths:
          - doc/
      script:
        - find . | grep -v ".git"
        - mkdir -p doc
        - touch doc/cache.txt

    testcache:
      type: test
      script:
        - env
        - find . | grep -v ".git"
        - tar xvzf ${CACHE_FILE}
        - ls doc/cache.txt

Perhaps this is due to how you reduced, but please heed "Don’t mix the caching with passing artifacts between stages. Caching is not designed to pass artifacts between stages." https://docs.gitlab.com/ee//ci/caching/index.html#cache-vs-artifacts — Jed, Jun 24 '19 at 16:58

score 23 · Answer 1 · edited Apr 13 '16 at 15:32

23

https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/327

image: java:openjdk-8-jdk

before_script:
    - export GRADLE_USER_HOME=`pwd`/.gradle

cache:
  paths:
    - .gradle/wrapper
    - .gradle/caches

build:
  stage: build
  script:
     - ./gradlew assemble

test:
  stage: test
  script:
     - ./gradlew check

edited Apr 13 '16 at 15:32

Stefan Profanter

6,458
6
41
73

answered Feb 18 '16 at 10:34

wcc526

3,915
2
31
29

2

I used the config above but the gradle wrapper was still downloading the files into .gradle/wrapper. Maybe it's some other related issue. – Stefan Profanter Apr 14 '16 at 07:29
1

I have to say I have seen the same results as @SailAvid. I also attempted to cache about every possible gradle directory that could exist from the first job (e.g. build) and all dependencies were getting pulled again in the second job. – user376327 Oct 01 '16 at 14:42
1

Same here. When reading the file, it looks like we're setting `GRADLE_USER_HOME` to a path *inside* the build-folder (so we can cache it) and the cache seems to be created and applied on following jobs but gradle keeps downloading. Any new ideas on this? – hardysim Mar 28 '17 at 08:58
Same happens to me, it saves the cache and applies it but Gradle is still downloading, any news on this? – nilsi Jan 03 '18 at 15:07

score 14 · Accepted Answer · edited Jan 24 '19 at 18:39

8.2 only supports per-job cache, and 8.3 will introduce "group" cache that serves among jobs according to @ayufan's comment in Possibility to cache folders in build directory (#97).

However, although I cannot be 100% sure, by quick digging the source code of gitlab-ci-muti-runner, docker executor doesn't seems to work with the cache functionality. Since a whole new container is created and destroyed in every job, the cache.tgz archive would no longer exist in the next build.

Errata:

The above description is incorrect due to testing in a misconfigured environment. By default, gitlab-ci-multi-runner creates a dedicated data volume container as a cache storage for each concurrent build. The cache container is mounted to directory /cache in the application container and those cache.tgz tarballs are placed under /cache by default. So caches are actually reusable among independent builds.

Updates 2015/12/11:

Just found out that "group" cache has already been implemented in gitlab-runner@7dc9524f6ef0144b3797fc07c9035f38a8ad0512, maybe not yet released and documented. You can enable it with

cache:
  paths:
    - doc/
  group: sharedcache

The result is one cache tarball being placed under path <namespace>/<repo>/sharedcache/ instead of two cache tarballs being placed separately under path <namespace>/<repo>/createcache/ and <namespace>/<repo>/testcache/.

Updates 2017/12/04:

"group" cache has been replaced by cache:key. Use the key key to make the cache share between jobs or git references. By default, a cache is shared between all jobs. So, simply write the following would do the job

cache:
  paths:
    - doc/

Checkout GitLab CI cache:key and gitlab-runner@d34f76f86a7c2fc4d399e9922175bcc614434016 for more information.

CI linter says `test:cache config contains unknown keys: group` — Kyslik, Dec 04 '17 at 09:06
@Kyslik It has been replaced by `cache:key`. Check out https://gitlab.com/gitlab-org/gitlab-runner/commit/d34f76f86a7c2fc4d399e9922175bcc614434016 and https://docs.gitlab.com/ee/ci/yaml/#cache-key — Lance Chen, Dec 04 '17 at 09:33
Just a comment - it's *much* easier to read your answer if you finish each edit with a complete answer. There is an edit history, so we can see what you've changed. There's no need for headings for edits; just update the content so it reads clearly. — Duncan Jones, Jun 20 '18 at 08:42

score 4 · Answer 3 · answered Apr 27 '17 at 11:53

4

It seems it is not possible to cache container local files with shared runners. You have to put your files in e.g. the cache folder:

before_script:
  - export GRADLE_USER_HOME=/cache/.gradle

...

cache:
  paths:
    - $GRADLE_USER_HOME/caches/
    - $GRADLE_USER_HOME/wrapper/
    - $GRADLE_USER_HOME/build-cache/

Strictly it is not required to put the /cache folder as files to cache, because this happens automatically, but I leave it for clarity (also when I want to move the gradle cache)

answered Apr 27 '17 at 11:53

Patrick

33,984
10
106
126

Is the `/cache` documented somewhere? I've only seen docs mentioning [cache:](https://docs.gitlab.com/ce/ci/yaml/#cache) – Alex Bitek Dec 17 '17 at 12:41
I remember it reading it somewhere, maybe they changed the behavior and/or removed it from the docs? – Patrick Jan 02 '18 at 14:51

score 0 · Answer 4 · answered Dec 09 '19 at 12:54

We found out, that is better to cache the whole .gradle-folder. This should be used only once per project. If k8s used for gitlab-runner, then you should turn the daemon off.

gitlab-ci.yaml:

variables:
  GRADLE_OPTS: "-Dorg.gradle.daemon=false" # no gradle-daemon on k8s

# ...

buildJar:
  stage: build
  image: gradle:5.6.4-jdk11
  before_script:
    - export GRADLE_USER_HOME=`pwd`/.gradle
  script:
    - gradle assemble
  cache:
    key: "gradleCache"
    paths:
      - .gradle

gitlab 8.2.1, How to use cache in .gitlab-ci.yml

Here is my .gitlab-ci.yml

Output of the job 'createcache'

Output of the job 'testcache'

My workaround

4 Answers4

Linked