how to prevent gitlab ci from downloading sbt every time?

Question

We have a play2/scala application which we are building with gitlab ci.

Our .gitlab-ci.yml (at least the important part) looks as follows:

image: hseeberger/scala-sbt

variables:
  SBT_GLOBAL_BASE_DIR: "$CI_PROJECT_DIR/cache/.sbt"
  IVY2_CACHE_DIR: "$CI_PROJECT_DIR/cache/.ivy2"
  SBT_BOOT_DIR:  "$CI_PROJECT_DIR/cache/.sbt/boot"
  M2_HOME_DIR: "$CI_PROJECT_DIR/cache/.m2"

before_script:
  # Log the sbt version
  - sbt sbt-version

build:
  stage: build
  script:
    - ./build.sh

with build.sh:

sbt -Dsbt.global.base=$SBT_GLOBAL_BASE_DIR \
  -Dsbt.ivy.home=$IVY2_CACHE_DIR \
  -Dsbt.boot.directory=$SBT_BOOT_DIR \
  compile

Unfortunately, our pipeline always runs for around 30-40 minutes with all the steps (build, verification, deploy). Most of the time it spends by downloading sbt over and over again what is really annoying.

I might not know enough about gitlab ci runners but from my understand, by using hseeberger/scala-sbt as the image, sbt should be globally available and there should be no need to download it.

Then also this solution from gitlab would not be necessary.

Anyhow, I would be glad if sbt would not be downloaded totally 6 times during each deployment whenever the server runs any sbt command.

Can someone explain me how to use the right image or the image in the right way or otherwise how I can cache the sbt stuff?

Update

Over the last days I fought a lot with docker and gitlab ci. I found that this problems is pretty much the same as described in don't downloading the internet. It seems that this is some hard task to have all the dependencies and should be best done by mounting them. That's unfortunately not possible as such on a shared gitlab ci runner.

I went on and discovered sbt-docker which allows you to build docker containers from a build.sbt file. With the package basic approach I tried to include all the locally available dependencies for the project into the container as global sbt plugins. But also this didn't help.

My last discovery was this answer regarding the maven solution and tried to translate that into our sbt project:

.gitlab-ci.yml

image: hseeberger/scala-sbt

variables:
  MAVEN_OPTS: -Dmaven.repo.local=/cache/maven.repository

stages:
  - build
  - test
  - staging
  - deploy

build:
  stage: build
  script:
    - sbt compile -Dsbt.ivy.home=/cache/.ivy2 -Dsbt.global.base=/cache/.sbt/0.13 -Dsbt.boot.directory=/cache/.sbt/boot -Dsbt.repository.config=/cache/.sbt/repositories

I can access the gitlab ci logs again. They look basically as follows:

[info] Loading project definition from /builds/kwiqjobs/backend/project
[info] Updating {file:/builds/kwiqjobs/backend/project/}backend-build...
[info] Resolving com.typesafe.play#sbt-plugin;2.5.4 ...

[info] Resolving com.typesafe.play#sbt-plugin;2.5.4 ...

[info] Resolving com.typesafe.play#sbt-routes-compiler_2.10;2.5.4 ...

[info] Resolving com.typesafe.play#sbt-routes-compiler_2.10;2.5.4 ...

[info] Resolving org.scala-lang#scala-library;2.10.6 ...

[info] Resolving com.typesafe.play#twirl-api_2.10;1.1.1 ...

[info] Resolving com.typesafe.play#twirl-api_2.10;1.1.1 ...

... a **lot** more

[info]  [SUCCESSFUL ] com.typesafe.sbt#sbt-twirl;1.1.1!sbt-twirl.jar (1033ms)
[info] downloading https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-native-packager/scala_2.10/sbt_0.13/1.0.3/jars/sbt-native-packager.jar ...
[info]  [SUCCESSFUL ] com.typesafe.sbt#sbt-native-packager;1.0.3!sbt-native-packager.jar (954ms)
[info] downloading https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-web/scala_2.10/sbt_0.13/1.3.0/jars/sbt-web.jar ...
[info]  [SUCCESSFUL ] com.typesafe.sbt#sbt-web;1.3.0!sbt-web.jar (1010ms)
[info] downloading https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-js-engine/scala_2.10/sbt_0.13/1.1.3/jars/sbt-js-engine.jar ...
[info]  [SUCCESSFUL ] com.typesafe.sbt#sbt-js-engine;1.1.3!sbt-js-engine.jar (1147ms)
[info] downloading https://repo1.maven.org/maven2/com/typesafe/play/twirl-api_2.10/1.1.1/twirl-api_2.10-1.1.1.jar ...
[info]  [SUCCESSFUL ] com.typesafe.play#twirl-api_2.10;1.1.1!twirl-api_2.10.jar (89ms)
[info] downloading https://repo1.maven.org/maven2/commons-io/commons-io/2.4/commons-io-2.4.jar ...
[info]  [SUCCESSFUL ] commons-io#commons-io;2.4!commons-io.jar (48ms)

a **lot** more
[info] Done updating.
[info] Compiling 228 Scala sources and 4 Java sources to /builds/kwiqjobs/backend/target/scala-2.11/classes...
[info] 'compiler-interface' not yet compiled for Scala 2.11.8. Compiling...
[info]   Compilation completed in 17.735 s
[success] Total time: 149 s, completed Jan 20, 2017 2:22:52 PM
Build succeeded

And I would like to get rid off all the downloading.

The solution from gitlab you linked to is not included in your build script, are you using it or not? Also please post the relevant parts of the output. — Jakub Kania, Jan 17 '17 at 23:03
No, it's not included. I am mostly wondering whether it should not be possible to avoid downloading `sbt` _at all_ with the hseeberger/scala-sbt image. On this docker image, `sbt` is already installed. Is there then any need to cache or install it? — stubbi, Jan 18 '17 at 08:55
It seems that I cannot access the logs right now, gitlab.com isn't responding. But you will see that after `sbt sbt-version` has been called, all `sbt` dependencies will be resolved and downloaded. Afterwards, when `build.sh` is being called, it will start all over again downloading them. Then, on `verification.sh` and `deploy.sh` **again** _everything_ will be downloaded again. Is the [gitlab solution](https://docs.gitlab.com/ee/ci/examples/test-scala-application.html#add-gitlab-ci-yml-file-to-project) the one I need to go with? Can I do some additional caching there? — stubbi, Jan 18 '17 at 08:59
You are downloading all the dependencies and building the application 3 times it seems. I added most of the options you can use. — Jakub Kania, Jan 18 '17 at 17:23
Please have a look at https://leonard.io/blog/2017/05/gitlab-ci-caching-for-sbt-projects/ on how to enable caching. I tested it, and it works as expected. — jurgispods, Sep 15 '17 at 06:29
ah, wow! We solved it now our way but it seems that that work. Thanks! Should we give it as an answer here and upvote it? — stubbi, Sep 15 '17 at 08:02
@stubbi You're right, I will prepare an answer (with proper recognition of the original source) — jurgispods, Sep 20 '17 at 19:25

jurgispods · Accepted Answer · 2019-03-16T20:17:10.217

If you don't want to use custom made images, the best solution is to use Gitlab CI's caching mechanism.

It's a little hard to get it right, but this blog post describes how to do it for SBT.

Example `.gitlab-ci.yml`

Quoted from the blog post, minor errors corrected by myself:

# some parts originally from https://github.com/randm-ch/units-of-information/blob/master/.gitlab-ci.yml

image: "hseeberger/scala-sbt"

variables:
  SBT_VERSION: "0.13.9"
  SBT_OPTS: "-Dsbt.global.base=sbt-cache/.sbtboot -Dsbt.boot.directory=sbt-cache/.boot -Dsbt.ivy.home=sbt-cache/.ivy"

cache:
  key: "$CI_BUILD_REF_NAME" # contains either the branch or the tag, so it's caching per branch
  untracked: true
  paths:
    - "sbt-cache/.ivy/cache"
    - "sbt-cache/.boot"
    - "sbt-cache/.sbtboot"
    - "sbt-cache/target"

stages:
  - test

test:
  script:
    - sbt test

Second example, also including `apt-get` caching

This is what I used for my project, usable for more general use cases and Docker images:

image: java:8

stages:
  - test

variables:
  SBT_VERSION: "0.13.9"
  SBT_OPTS: "-Dsbt.global.base=sbt-cache/.sbtboot -Dsbt.boot.directory=sbt-cache/.boot -Dsbt.ivy.home=sbt-cache/.ivy"
  SBT_CACHE_DIR: "sbt-cache/.ivy/cache"

cache:
  key: "$CI_BUILD_REF_NAME" # contains either the branch or the tag, so it's caching per branch
  untracked: true
  paths:
    - "apt-cache/"
    - "sbt-cache/.ivy/cache"
    - "sbt-cache/.boot"
    - "sbt-cache/.sbtboot"
    - "sbt-cache/target"

before_script:
  - export APT_CACHE_DIR=`pwd`/apt-cache
  - mkdir -pv $APT_CACHE_DIR
  - ls $APT_CACHE_DIR || echo "no apt-cache dir found"
  - apt-get -o dir::cache::archives=$APT_CACHE_DIR update -y
  - apt-get -o dir::cache::archives=$APT_CACHE_DIR install apt-transport-https -y
  # Install SBT
  - mkdir -pv $SBT_CACHE_DIR
  - ls $SBT_CACHE_DIR || echo "no ivy2 cache fir found"
  - echo "deb http://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list
  - apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
  - apt-get -o dir::cache::archives=$APT_CACHE_DIR update -y
  - apt-get -o dir::cache::archives=$APT_CACHE_DIR install sbt -y
  - sbt -v sbtVersion

test:
  stage: test
  script:
     - sbt -v sbtVersion

score 2 · Answer 2 · edited May 23 '17 at 12:24

2

There are 4 things you can do:

own docker image
cache
artifacts
use own caching solution

The best solution will be a mix of them all.

You can build your own docker image with all the dependencies you need, it's the fastest solution as it won't have to download everything but it will introduce another piece of the puzzle you need to take care of. You can use the in-built gitlab repository for storing it and have gitlab build it and then use it.
You can use cache in Gitlab CI jobs so it won't have to download everything all the time. The default cache for sbt seems to be ~/.ivy2 so add

cache:
  paths:
    - ~/.ivy2/

As the first lines in your gitlab file to use the caches for each stage.

Define the target directory as the artifact to pass it between builds and so you don't have to build it on each stage.

artifacts:
  paths:
    - target/

You can store the files you need to cache in your own s3/minio/nfs if the options supplied by gitlab are not enough. This will be a very custom solution so you will have to find your own way around it.

edited May 23 '17 at 12:24

Community

1
1

answered Jan 18 '17 at 17:20

Jakub Kania

15,665
2
37
47

Thanks so far! I will test them. But regarding 1: Why do I need to create my own docker image if the one I am using from the public repository should have already the sbt dependency which it downloads during the build anyway? Why is that not working? – stubbi Jan 19 '17 at 08:29
@stubbi Well, I'm not familiar with stb but I think it's pulling the dependencies of your project or self updating. At least it looked like that when i tried running an example project. – Jakub Kania Jan 19 '17 at 09:21
I found the problem. The sbt version in the container and in our project must match! We where using `0.13.11` while in the container there is `0.13.13`. That speeds up everything but there is still the dependencies which should be in the container then. And at best they should also always be up to date. I found [this](http://stackoverflow.com/questions/34384657/how-to-run-a-sbt-project-in-docker-by-using-sbt-docker-or-writing-a-docker-file) question here with a hint on [sbt native packager docker plugin](http://www.scala-sbt.org/sbt-native-packager/formats/docker.html) and will try that now. – stubbi Jan 19 '17 at 12:53
I updated the question. Might you have a look again? Maybe you can see better now where the problem is. – stubbi Jan 20 '17 at 14:38

how to prevent gitlab ci from downloading sbt every time?

Update

.gitlab-ci.yml

2 Answers2

Example `.gitlab-ci.yml`

Second example, also including `apt-get` caching

Linked

how to prevent gitlab ci from downloading sbt every time?

Update

.gitlab-ci.yml

2 Answers2

Example .gitlab-ci.yml

Second example, also including apt-get caching

Linked

Example `.gitlab-ci.yml`

Second example, also including `apt-get` caching