102

We have a project that is composed of multiple (non-public) repositories.

To build the whole project, the build system needs to have the files of all repositories (master branches).

Is there a way I can configure GitLab CI to provide the repositories I need?

I guess I could do a git fetch or similar during the CI build, but how to deal with authentication then?

vvvvv
  • 25,404
  • 19
  • 49
  • 81
Udo G
  • 12,572
  • 13
  • 56
  • 89

4 Answers4

253

If you are running gitlab version 8.12 or later, the permissions model was reworked. Along with this new permission model comes the the CI environment variable CI_JOB_TOKEN. The premium version of GitLab uses this environment variable for triggers, but you can use it to clone repos.

dummy_stage:
  script:
    - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.instance/group/project.git
mkobit
  • 43,979
  • 12
  • 156
  • 150
microe14
  • 2,711
  • 1
  • 15
  • 5
  • 11
    Note that, the as is correctly documented in the example above, within the GitLab url the separator between the instance and the group must be a slash (...gitlab.instance/group/project.git) and **not** a colon (...gitlab.instance:group/project.git) in order for the clone to succeed. The SSH url for a repo takes the latter form which caused be to use mistakenly use that format. – Grant Humphries Jul 15 '19 at 23:46
  • SSH Keys/Tokens would be no hack if you need to fetch them from other GL instances. On the other hand you could create imported forks in your local instance of these repos and (use another repo with sync scripts to) sync them every now and then. – Ritualmaster Aug 19 '19 at 10:30
  • Deploy keys are not a hack, agreed. My answer is going for least privileges to accomplish the task (and also not have to change any configuration). The up side of deploy keys is that it opens up a wide range of possibilities (write access, repos outside of your gitlab instance, etc). – microe14 Aug 20 '19 at 17:43
  • 1
    @GrantHumphries, thanks for the tip there. I just made that mistake and was pulling out my hair trying to figure out what was wrong – redsoxfantom Jan 13 '21 at 16:41
  • 1
    This answer should be accepted because it really answer to the question – jmcollin92 Mar 02 '22 at 17:05
  • also you can use `$CI_REGISTRY_USER` in user section. It is the same as "gitlab-ci-token" – gam6itko Mar 25 '23 at 11:10
  • The destination repository (being cloned) should have the source repository (running pipeline) in the allowlist for the CI_JOB_TOKEN to get access to it. More details here : https://docs.gitlab.com/ee/ci/jobs/ci_job_token.html#add-a-project-to-the-job-token-scope-allowlist – hassanbsalimi May 24 '23 at 02:24
  • This makes it a pain to use the local gitlab-runner. – jeremyjjbrown Jun 02 '23 at 15:49
  • I'm new to CI. I was wondering what the behavior is when you use the clone command if your repo has already been cloned to the runner. Does it default to fetch? – tzg Aug 29 '23 at 13:29
30

A couple of workarounds (I hate this word!) that work-around-ed for me:

  1. Using git submodule, see https://docs.gitlab.com/ce/ci/git_submodules.html

  2. Re-using $CI_REPOSITORY_URL defined by Gitlab and available even inside child Docker containers. This env var already contains username and password, that can be used for another repo on the same server. See snippet from .gitlab-ci.yml:

- BASE_URL=`echo $CI_REPOSITORY_URL | sed "s;\/*$CI_PROJECT_PATH.*;;"`
- REPO_URL="$BASE_URL/thirdparty/gtest.git"
- REPO_DIR=thirdparty/gtest
- rm -fr $REPO_DIR
- git clone $REPO_URL $REPO_DIR
  1. Even storing that URL with username\password in ~/.git-credentials file and configuring git to use it via credential.helper. All further "git clone" commands will use it.
- echo Storing git credentials to be used by "git clone" commands without username and password ...
- GIT_CREDENTIALS_FILE=~/.git-credentials
- BASE_URL=`echo $CI_REPOSITORY_URL | sed "s;\/*$CI_PROJECT_PATH.*;;"`
- echo $BASE_URL > $GIT_CREDENTIALS_FILE
- git config --global credential.helper store --file=$GIT_CREDENTIALS_FILE

HOWEVER !

Having spent quite a few years in CI \ CD field, I don't think it's a good design that requires linking repositories as sources.

Yes, in classic CI tools like Jenkins or TeamCity you can create a job that fetches several Git repos in different subdirectories.

But I like GitLab CI way of Pipeline As Code, where .gitlab-ci.yml controls the build of this very repo and you don't even have to think about that whole pre-build step of getting sources. Then such build would publish binary artifacts and downstream projects\repos can use those instead of sources of dependencies. It's also faster.

Separation of concerns.

I don't there is an official way in my .gitlab-ci.yml to use artifacts of another project. But there are other ways like hooks, Gitlab API, though such bespoke solutions require maintenance.

There's better way - to publish\fetch artifacts to\from external widely-adopted Package Manager. Depending on your language it could be Maven, NuGet, npm, jFrog Artifactory, Nexus, etc. Another advantage of this method is that developers can follow the same process in their local builds, which is not easily done if dependencies are defined in .gitlab-ci.yml

It's a bigger problem for native code (Cxx) mainly due to Binary Interface compatibility, but things like Conan.io etc are slowly catching up.

Ivan
  • 9,089
  • 4
  • 61
  • 74
  • Upvoted for solution 2. My variant looks like `git clone $(echo $CI_REPOSITORY_URL | sed -e s"|$CI_PROJECT_PATH|$SECOND_PROJECT_PATH|")` – EdwardTeach Jun 23 '21 at 19:59
  • Currently there is an official way to get [artifact from another project](https://docs.gitlab.com/ee/ci/yaml/#needsproject), but it require premium. – Tudax Feb 10 '22 at 09:01
  • 1
    Coming originally from Java, I understand the desire to use binaries instead of source. I recently started working in Go though where code dependencies are pulled from source. In this case, we need to be able to link to the sources. Go lets you go to a tag, branch or commit SHA so it is at least still versioned. – Matthew Sandoz Jul 10 '22 at 17:13
8

You can add a deploy key to all projects. Then configure the deploy key's private key on the runner(s). Use normal git commands in your build process to clone the repositories on the runner. This may require a bit of configuration on your runners, but it should work.

You can either generate one SSH key pair and use that on all runners or generate one key pair per runner. To generate an SSH key pair follow SSH key documentation. The private key should be placed in 'gitlab-runner' user's .ssh directory so the git command can present it at clone time. The public key should be added to the GitLab project as a deploy key. Add a deploy key in the project settings -> 'Deploy Keys'.

Drew Blessing
  • 2,595
  • 11
  • 16
  • could you please be a bit more precise what you mean with *"Then configure the deploy key's private key on the runner(s)."* ? – Udo G Oct 08 '15 at 13:28
  • Edit: I updated the answer with more details. Let me know if you still have questions. – Drew Blessing Oct 08 '15 at 20:40
  • 3
    Maybe things have changed since 2015, but on Gitlab.com, it seems like the runner runs stuff as `root`. And there is no `gitlab-runner` user in my case. – conradkleinespel Dec 10 '16 at 22:47
  • Biggest problem with ssh/keys is that it allows the user of the runner (the writer of the gitlab file to be able to access and copy out that ssh key). – CMCDragonkai Dec 16 '19 at 02:06
  • Like me, if you are hearing about deploy keys/tokens for the first time, [here](https://docs.gitlab.com/ee/user/project/deploy_keys/) is the link to the documentation. It has examples and is very easy to understand. – Isuru Jul 28 '21 at 14:51
-1

In my use case, I am pulling git sub-modules with LFS storage; in this case, this is what worked best:

sed -i "s%url = git@gitlab.com:%url = https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/%g" .git/config .gitmodules
mangelajo
  • 1