How to use Gitlab CI to build a Java Maven project?

Question

I have been experimenting with no success whatsoever, I am running a Gitlab hosted on Linux, and trying to get my head around the CI functionality.

According to the Gitlab documentation you only need to create a .gitlab-ci.yml file, the Gitlab implementation of Travis-CI. Now from the looks of it you can accomplish a lot with the .gitlab-ci.yml, but a lot of the documentation is referencing Ruby and other languages. Nothing is said about how to build Java Maven projects.

How can I build a simple application in Java? Can I use the shared runner, or should I be using a specific runner, in that case what or which runner implementation should I choose: ssh, docker, or shell? Then, what should I put in the .gitlab-ci.yml file at least to build the project with Maven?

Answer 1

Register a Docker runner and use one of the official Maven Docker images, e.g., maven:3-jdk-11 in your .gitlab-ci.yml file:

image: maven:3-jdk-11

build:
  script: "mvn install -B"

Note the -B flag, which is recommended for non-interactive use.

As far as I understand, it does not matter whether the runner is shared or specific.

Answer 2

Several questions here.

I'll start answering the Java build question, then the Runners one.

Java build

I'll start from the most basic Java build configuration, and progressively add features.

1. basic Java build

This configuration will hopefully run your Maven build (and only the build, explicitly excluding the unit tests):

stages:
  - build

java-build:
  # select the most appropriate image for your project
  image: maven:3.8-openjdk-11
  stage: build
  script:
    - mvn package -DskipTests=true

2. with artifacts, cache and recommended Maven options

This new version:

• declares Maven build output as GitLab artifacts (for later use in the downstream pipeline),
• takes benefit of GitLab's cache to cache local Maven repository (in .m2/repository),
• also enforces some recommended Maven options to use in CI/CD context.

stages:
  - build

variables:
  # This will suppress any download for dependencies and plugins or upload messages which would clutter the console log.
  # `showDateTime` will show the passed time in milliseconds. You need to specify `--batch-mode` to make this work.
  MAVEN_OPTS: >-
    -Dhttps.protocols=TLSv1.2
    -Dmaven.repo.local=.m2/repository
    -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN
    -Dorg.slf4j.simpleLogger.showDateTime=true
    -Djava.awt.headless=true
  # As of Maven 3.3.0 instead of this you may define these options in `.mvn/maven.config` so the same config is used
  # when running from the command line.
  # `installAtEnd` and `deployAtEnd` are only effective with recent version of the corresponding plugins.
  MAVEN_CLI_OPTS: >-
    --batch-mode
    --errors
    --fail-at-end
    --show-version
    -DinstallAtEnd=true
    -DdeployAtEnd=true  

java-build:
  # select the most appropriate image for your project
  image: maven:3.8-openjdk-11
  stage: build
  # Cache downloaded dependencies and plugins between builds.
  # The key here separates one cache per branch/tag ($CI_COMMIT_REF_SLUG)
  cache:
    key: "maven-$CI_COMMIT_REF_SLUG"
    paths:
      - .m2/repository
  script:
    - mvn $MAVEN_CLI_OPTS package -DskipTests=true
  artifacts:
    name: "Maven artifacts from $CI_PROJECT_NAME on $CI_COMMIT_REF_SLUG"
    paths:
      - "**/target"

3. with unit tests

There are two options when integrating you unit tests in your CI/CD pipeline:

1. run them in the same job as the build
2. run them in a separate job

As a matter of pipeline execution speed and green-IT considerations, I definitely prefer option 1, but I admit people could prefer the second one.

Here is the new version of the .gitlab-ci.yml file, also implementing GitLab unit tests integration:

stages:
  - build

variables:
  # This will suppress any download for dependencies and plugins or upload messages which would clutter the console log.
  # `showDateTime` will show the passed time in milliseconds. You need to specify `--batch-mode` to make this work.
  MAVEN_OPTS: >-
    -Dhttps.protocols=TLSv1.2
    -Dmaven.repo.local=.m2/repository
    -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN
    -Dorg.slf4j.simpleLogger.showDateTime=true
    -Djava.awt.headless=true
  # As of Maven 3.3.0 instead of this you may define these options in `.mvn/maven.config` so the same config is used
  # when running from the command line.
  # `installAtEnd` and `deployAtEnd` are only effective with recent version of the corresponding plugins.
  MAVEN_CLI_OPTS: >-
    --batch-mode
    --errors
    --fail-at-end
    --show-version
    -DinstallAtEnd=true
    -DdeployAtEnd=true  

java-build-and-test:
  # select the most appropriate image for your project
  image: maven:3.8-openjdk-11
  stage: build
  # Cache downloaded dependencies and plugins between builds.
  # The key here separates one cache per branch/tag ($CI_COMMIT_REF_SLUG)
  cache:
    key: "maven-$CI_COMMIT_REF_SLUG"
    paths:
      - .m2/repository
  script:
    # the 'verify' goal is definitely the most appropriate here
    - mvn $MAVEN_CLI_OPTS verify
  artifacts:
    name: "Maven artifacts from $CI_PROJECT_NAME on $CI_COMMIT_REF_SLUG"
    paths:
      - "**/target"
    reports:
      # declare the JUnit reports (recursive pattern for multi-module projects)
      junit:
        - "**/target/*-reports/TEST-*.xml"

4. going further

From this step, the build job could still be enhanced further, for instance with code coverage computation and integration, but that requires a little more code.

Another way of implementing state-of-the-art pipeline with much less efforts is using GitLab CI/CD template. For example:

• GitLab's Maven template
• or to-be-continuous template

to be continuous is an open-source project that provides a collection of ready-to-use, configurable, extensible, composable templates.

About Runners

GitLab architecture is very versatile, with the notion of Runners. Runners are the basic executors pool that make it possible to execute GitLab CI/CD jobs.

2 things of interest to know about runners

1. you can specialise your runners

With GitLab, you can register several kind of runners, made for special and complementary purpose.

In order to segregate them, GitLab supports the notion of Tags. When registering your runners, you shall associate them with functional tag names, that will help developers select the most appropriate one in their .gitlab-ci.yml file.

For example, let's imagine you have 4 runners:

#	Description	Proposed Tags
1	Linux-based general purpose runners for building code, running tests, ... with transparent access to the internet	linux, general, internet you should also allow this one to run untagged jobs (makes it kind of default runner)
2	Microsoft-based general purpose runners for building your .NET code	windows, general, internet
3	compute-optimized runners, made for - say - training super-secret neural networks with no access to the internet	linux, compute, ml (for machine learning)
4	runners located behind your DMZ in your on-premises datacenter, used to perform code/infrastructure deployments	linux, deploy, datacenter

With this setup, a monorepo project with both Java and .NET code may declare the following .gitlab-ci.yml file:

stages:
  - build
  - test
  - deploy

# this job declares no tag: will be executed by default runner (#1)
java-build:
  image: maven:3.8-openjdk-11
  stage: build
  script:
    - Java build code here

dotnet-build:
  image: mcr.microsoft.com/dotnet/sdk:5.0
  stage: build
  tags:
    - windows
    - general
  script:
    - .NET build code here

# this job declares no tag: will be executed by default runner (#1)
java-test:
  image: maven:3.8-openjdk-11
  stage: test
  script:
    - Java test code here

dotnet-test:
  image: mcr.microsoft.com/dotnet/sdk:5.0
  stage: test
  tags:
    - windows
    - general
  script:
    - .NET test code here

deploy:
  stage: deploy
  tags:
    - deploy
    - datacenter
  script:
    - deployment code here

2. runners have different scopes

Quoting the official documentation:

Runners are available based on who you want to have access:

• Shared runners are available to all groups and projects in a GitLab instance.
• Group runners are available to all projects and subgroups in a group.
• Specific runners are associated with specific projects. Typically, specific runners are used for one project at a time.

Answer 3

I would like to add bit of information here guys. First let's clear out some confusion regarding shared and specific runner.

Shared Runner: As its name sound, shared runners are the build process flow instances which can be used to execute jobs of every single project in your installed gitlab instance having Allowed Shared runners option enabled. To do that of-course you would need administrative rights. As per current gitlab documentation, only use with administrative rights is able to define shared runner.

specific runner This kind of runner executes jobs of only one project.

Also, these are few important points to keep in mind when choosing runner for your projects.

1. Shared Runners are useful for jobs that have similar requirements, between multiple projects. Rather than having multiple runners idling for many projects, you can have a single or a small number of runners that handle multiple projects. This makes it easier to maintain and update runners for common set of projects.
2. Specific runners are useful for jobs that have special requirements or for projects with a specific demand. If a job has certain requirements, you can set up the specific runner with this in mind, while not having to do this for all runners. For example, if you want to deploy a certain project, you can setup a specific runner to have the right credentials for this.

Now to select right executor for project, its very important that we have bird view on all the available executors for gitlab runner. Gitlab has made this job easy for us by providing nice documentation over here explaining what are the different options you will get with different executors.

If you would like to know more about runners and different executors, I would suggest you start with this article, Gitlab Runner

Answer 4

I spent a fair amount of time trying to set up our Java projects on Gitlab CI. I got it working with some degree of success. As mentioned by rolve, the most straight forward solution is to use an image from the official repo: https://hub.docker.com/_/maven

However, we have a corporate proxy which was causing my builds to get timeout requests when fetching the project's dependencies. I tried many solutions and finally came across this post: https://gitlab.com/gitlab-org/gitlab-ce/issues/15167.

The post itself is about setting up maven to cache downloaded dependencies in a local repo which can be accessed among builds. The idea is that you can write a local maven configuration file by in .gitlab-ci.yml to set up your cache directory and your proxy.

before_script:
  -echo '<settings
          xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
          https://maven.apache.org/xsd/settings-1.0.0.xsd">
          <localRepository>/cache/.m2</localRepository>
          <proxies>
              <proxy>
                  <active>true</active>
                  <protocol>'$PROXY_PROTOCOL'</protocol>
                  <host>'$PROXY_HOST'</host>
                  <port>'$PROXY_PORT'</port>
              </proxy>
          </proxies>
      </settings>' > $HOME/.m2/settings.xml

build_debug1:
  stage: build
  script: "echo $PROXY_HOST"

build_debug2:
  stage: build
  script: "cat $HOME/.m2/settings.xml"

build_maven:
  stage: build
  script: "mvn $MAVEN_CLI_OPTS package"
  artifacts:
    paths:
      - target/*.jar

deploy_debug1:
  stage: package
  script: "ls target/"

Notice the build debug jobs are only to see whether the proxy settings were being correctly injected. You can set the proxy environment variables as secrets using Gitlab by going to Project -> Settings -> CI/CD Pipelines -> Secret Variables.

The last deploy_debug job is to see what was generated in your target directory.

Answer 5

The documentation describes the YAML syntax used to control builds:

• https://gitlab.com/help/ci/yaml/README.md

So why don't you try starting with the following?:

job1:
  script: "mvn package"

Presumably this will only work if Maven is already installed, so you'll need a runner that supports this.

I have not used GitLab but the documentation suggests you could further customize it to use the official Maven Docker image to perform the builds. Looks very interesting, but I'd agree documentation is missing a Java example.

Answer 6

I use this command but in general the documentation on java/maven builds seems quite rare

maven-package:
  script: "mvn install -B"

Answer 7

With the help of How to deploy Maven projects to Artifactory with GitLab CI/CD

You can compile your java maven project by just adding a .gitlab-ci.yml file to your repository's root directory with the following content:

image: maven:latest

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"

cache:
  paths:
    - .m2/repository/
    - target/

build:
  stage: build
  script:
    - mvn compile

Answer 8

Just checking if it is allowed or to add steps after mvn build/test command. I am checking if there is an option to run script after build irrespective of build status.

some thing like below. where I want to run irrespective of mvn test state. any thoughts? I know one way to run it as after-script. but trying to avoid that.

build:
  stage: build
  script:
    - mvn test
    - ./extract_data.sh

Answer 9

#-----------Base image that contains required maven and jdk--------#
image: basebuildimage
variables:
  MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
stages:
   - build
#----------Build code-----------#
build:
  stage: build  
  script:
    - mvn $MAVEN_CLI_OPTS clean install