18

Question

  • What is the best way to carry artifacts (jar, class, war) among projects when using docker containers in CI phase.

Let me explain my issue in details, please don't stop the reading... =)

Gitlabs project1

  • unit tests
  • etc...
  • package

Gitlabs project2

  • unit test
  • etc...
  • build (failing)
    • here I need one artifact (jar) generated in project1

Current scenario / comments

  • I'm using dockers so in each .gitlab-ci.yml I'll have independent containers
  • All is working fine in project1
  • If I use "shell" instead of dockers in my .gitlab-ci.yml I can keep the jar file from the project1 in the disk and use when project2 kicks the build
  • Today my trigger on call project2 when project1 finish is working nicely
  • My artifact is not an RPM so I'll not add into my repo

Possible solutions

Community
  • 1
  • 1
rafa.ferreira
  • 1,997
  • 7
  • 26
  • 41

5 Answers5

13

In GitLab silver and premium, there is the $CI_JOB_TOKEN available, which allows the following .gitlab-ci.yaml snippet:

build_submodule:
  image: debian
  stage: test
  script:
  - apt update && apt install -y unzip
  - curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN"
  - unzip artifacts.zip
  only:
  - tags

However, if you do not have silver or higher gitlab subscriptions, but rely on free tiers, it is also possible to use the API and pipeline triggers.

Let's assume we have project A building app.jar which is needed by project B.

First, you will need an API Token. Go to Profile settings/Access Tokens page to create one, then store it as a variable in project B. In my example it's GITLAB_API_TOKEN.

In the CI / CD settings of project B add a new trigger, for example "Project A built". This will give you a token which you can copy. Open project A's .gitlab-ci.yaml and copy the trigger_build: section from project B's CI / CD settings trigger section.

Project A:

trigger_build:
  stage: deploy
  script:
    - "curl -X POST -F token=TOKEN -F ref=REF_NAME https://gitlab.example.com/api/v4/projects/${PROJECT_B_ID}/trigger/pipeline"

Replace TOKEN with that token (better, store it as a variable in project A -- then you will need to make it token=${TRIGGER_TOKEN_PROJECT_B} or something), and replace REF_NAME with your branch (e.g. master).

Then, in project B, we can write a section which only builds on triggers and retrieves the artifacts.

Project B:

download:
  stage: deploy
  only:
    - triggers
  script:
    - "curl -O --header 'PRIVATE-TOKEN: ${GITLAB_API_TOKEN}' https://gitlab.example.com/api/v4/projects/${PROJECT_A_ID}/jobs/${REMOTE_JOB_ID}/artifacts/${REMOTE_FILENAME}"

If you know the artifact path, then you can replace ${REMOTE_FILENAME} with it, for example build/app.jar. The project ID can be found in the CI / CD settings.

I extended the script in project A to pass the remaining information as documented in the trigger settings section:

Add variables[VARIABLE]=VALUE to an API request. Variable values can be used to distinguish between triggered pipelines and normal pipelines.

So the trigger passes the REMOTE_JOB_ID and the REMOTE_FILENAME, but of course you can modify this as you need it:

curl -X POST \
     -F token=TOKEN \
     -F ref=REF_NAME \
     -F "variables[REMOTE_FILENAME]=build/app.jar" \
     -F "variables[REMOTE_JOB_ID]=${CI_JOB_ID}" \
     https://gitlab.example.com/api/v4/projects/${PROJECT_B_ID}/trigger/pipeline
Sebastian Höffner
  • 1,864
  • 2
  • 26
  • 37
7

Hello you must take a look at a script named get-last-successful-build-artifact.sh and developed by morph027.

https://gitlab.com/morph027/gitlab-ci-helpers

This script allow to download an artifact and unzip it in the project root. It use Gitlab API to retrieve latest successful build and download corresponding artifact. You can combine multiple artifacts and unzip wherever you want just by updating the script a little.

I'm also currently starting a PHP library to handle build artifacts but it's in a very early stage and tied with laravel for the moment.

For the moment there is no easy way to handle artifact usage between projects, you must build your own using that tools.

I think using shell executor is not the right solution, it's very dangerous because you can't verify the file on the server used during the build !

Hope this help :)

shulard
  • 593
  • 5
  • 16
  • 1
    I marked your answer as the official one as your script helped on exactly what we need. Thanks! – rafa.ferreira Sep 23 '16 at 22:19
  • is it possible to use get-last-successful-build-artifact.sh *without* private-token (in a world-readable repository)? e.g. to share an artifact download command w/o exposing your token – pseyfert Feb 16 '17 at 21:13
  • For the moment it's not possible to retrieve artifact without authentication... To download the file you need to access Gitlab API which is private... – shulard Feb 17 '17 at 07:10
3

arry artifacts (jar, class, war) among projects

That should be what the package Registry is for.

With GitLab 13.3 (August 2020), it is now available for free!

Package Registry now available in Core

A year and a half ago, we expanded our support for Java projects and developers by building Maven support directly into GitLab. Our goal was to provide a standardized way to share packages and have version control across projects.

Since then, we’ve invested to build out the Package team further while working with our customers and community to better understand your use cases. We also added support for Node, C#/.NET, C/C++, Python, PHP, and Go developers.

Your increased adoption, usage, and contributions to these features have allowed us to expand our vision to a more comprehensive solution, integrated into our single application, which supports package management for all commonly-used languages and binary formats.
This goal could not have been achieved without the explicit support of the GitLab community.

As part of GitLab’s stewardship promises, we are excited to announce that the basic functionality for each package manager format is now available in the GitLab Core Edition.
This means that if you use npm, Maven, NuGet, Conan, PyPI, Composer or Go modules, you will be able to:

  • Use GitLab as a private (or public) package registry
  • Authenticate using your GitLab credentials, personal access, or job token
  • Publish packages to GitLab
  • Install packages from GitLab
  • Search for packages hosted on GitLab
  • Access an easy-to-use UI that displays package details and metadata and allows you to download any relevant files
  • Ensure that your contributions are available for ALL GitLab users

We look forward to hearing your feedback and continuing to improve these features with all of our users.

See Documentation and Issue.

See this video.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • While I am currently trying out this solution, one problem with this is that the package registry requires version numbers of the format x.y.z (see https://docs.gitlab.com/ee/user/packages/generic_packages/index.html#publish-a-package-file) and feels more like it's supposed for (permanently) publishing packages and not for sharing snapshots/artifacts for downstream depending builds. For example, if I have a UI and a backend and build both of them in individual repositories but merge them together into one package in a third repository, the UI and backend repos might be better of with artifacts – Sebastian Höffner Feb 26 '21 at 17:01
  • In fact, using the same version twice lists the packages inside the same page, as they are considered equal, thus I need to provide a proper version number per build/commit (and it's not even possible to use -rc1 or -beta/-snapshot/-dev suffixes. Thus, passing artifacts for "across-project-pipelines" might still be a viable solution. – Sebastian Höffner Feb 26 '21 at 17:02
  • @SebastianHöffner Good points. Re-reading https://gitlab.com/groups/gitlab-org/-/epics/4209 (from which that feature was specified), that package registry might not be the best fit for your current workflow. – VonC Feb 26 '21 at 17:03
2

Cool, found my snippet being referenced here ;)

is it possible to use get-last-successful-build-artifact.sh without private-token (in a world-readable repository)? e.g. to share an artifact download command w/o exposing your token

Yes, just add it as a secret variable in project settings -> pipelines -> secret variables.

morph027
  • 21
  • 1
0

As of this writing artifacts cannot be shared across project only within the pipeline. See https://docs.gitlab.com/ee/ci/yaml/README.html#artifacts

However there is an open feature to enable this facility which is not yet implemented. https://gitlab.com/gitlab-org/gitlab-ce/issues/14728

Kalpa Gunarathna
  • 1,047
  • 11
  • 17