Getting GitLab CI to clone private repositories

Question

I have GitLab & GitLab CI set up to host and test some of my private repos. For my composer modules under this system, I have Satis set up to resolve my private packages.

Obviously these private packages require an ssh key to clone them, and I have this working in the terminal - I can run composer install and get these packages, so long as I have the key added with ssh-add in the shell.

However, when running my tests in GitLab CI, if a project has any of these dependencies the tests will not complete as my GitLab instance needs authentication to get the deps (obviously), and the test fails saying Host key verification failed.

My question is how do I set this up so that when the runner runs the test it can authenticate to gitlab without a password? I have tried putting a password-less ssh-key in my runners ~/.ssh folder, however the build wont even add the key, "eval ssh-agent -s" followed by ssh-add seems to fail saying the agent isn't running...

Answer 1

See also other solutions:

• git submodule permission (see Marco A.'s answer)
• job token and override repo in git config (see a544jh's answer)

---

Here a full howto with SSH keys:

General Design

• generating a pair of SSH keys
• adding the private one as a secure environment variable of your project
• making the private one available to your test scripts on GitLab-CI
• adding the public one as a deploy key on each of your private dependencies

Generating a pair of public and private SSH keys

Generate a pair of public and private SSH keys without passphrase:

ssh-keygen -b 4096 -C "<name of your project>" -N "" -f /tmp/name_of_your_project.key

Adding the private SSH key to your project

You need to add the key as a secure environment variable to your project as following:

• browse https://<gitlab_host>/<group>/<project_name>/variables
• click on "Add a variable"
• fill the text field Key with SSH_PRIVATE_KEY
• fill the text field Value with the private SSH key itself
• click on "Save changes"

Exposing the private SSH key to your test scripts

In order to make your private key available to your test scripts you need to add the following to your .gitlab-ci.yml file:

before_script:
  # install ssh-agent
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
  # run ssh-agent
  - eval $(ssh-agent -s)
  # add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  # disable host key checking (NOTE: makes you susceptible to man-in-the-middle attacks)
  # WARNING: use only in docker container, if you use it with shell you will overwrite your user's ssh config
  - mkdir -p ~/.ssh
  - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config

Code Snippet comes from GitLab documentation

Adding the public SSH key as a deploy key to all your private dependencies

You need to register the public SSH key as deploy key to all your private dependencies as following:

• browse https://<gitlab_host>/<group>/<dependency_name>/deploy_keys
• click on "New deploy key"
• fill the text field Title with the name of your project
• fill the text field Key with the public SSH key itself
• click on "Create deploy key"

Answer 2

If you don't want to fiddle around with ssh keys or submodules, you can override the repo in git's configuration to authenticate with the job token instead (in gitlab-ci.yml):

before_script:
  - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com/group/repo.git".insteadOf git@gitlab.example.com:group/repo.git

Answer 3

I'm posting this as an answer since others weren't completely clear and/or detailed IMHO

Starting from GitLab 8.12+, assuming the submodule repo is in the same server as the one requesting it, you can now:

1. Set up the repo with git submodules as usual (git submodule add git@somewhere:folder/mysubmodule.git)

2. Modify your .gitmodules file as follows

    [submodule "mysubmodule"]
      path = mysubmodule
      url = ../../group/mysubmodule.git
   

   where ../../group/mysubmodule.git is a relative path from your repository to the submodule's one.

3. Add the following lines to gitlab-ci.yml

    variables:
      GIT_SUBMODULE_STRATEGY: recursive
   

   to instruct the runner to fetch all submodules before the build.

Caveat: if your runner seems to ignore the GIT_SUBMODULE_STRATEGY directive, you should probably consider updating it.

(source: https://docs.gitlab.com/ce/ci/git_submodules.html)

Answer 4

The currently accepted answer embeds Gitlab-specific requirements into my .gitmodules file. This forces a specific directory layout for local development and would complicate moving to another version control platform.

Instead, I followed the advice in Juddling's answer. Here's a more complete answer.

My .gitmodules files has the following contents:

[submodule "myproject"]
    url = git@git.myhost.com:mygroup/myproject.git

In my gitlab-ci.yml I have the following:

build:
  stage: build
  before_script:
    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@git.myhost.com/".insteadOf "git@git.myhost.com:"
    - git submodule sync && git submodule update --init

The trailing / and : are critical in the git config line, since we are mapping from SSH authentication to HTTPS. This tripped me up for a while with "Illegal port number" errors.

I like this solution because it embeds the Gitlab-specific requirements in a Gitlab-specific file, which is ignored by everything else.

Answer 5

I used deploy tokens to solve this issue, as setting up SSH keys for a test runner seems a little long winded.

git clone http://<username>:<deploy_token>@gitlab.example.com/tanuki/awesome_project.git

The deploy tokens are per project and are read only.

Answer 6

One way to solve this without changing the git repository's structure is to perform the following steps:

1. get ssh host keys

Get the ssh host keys of the server that you are running on. For gitlab.com:

1. run ssh-keyscan gitlab.com > known_hosts
2. check that ssh-keygen -lf known_hosts agrees with the fingerprints reported here.
3. copy the content of the known_hosts and paste it on a variable called SSH_KNOWN_HOSTS on the repository.

This step is only needed once.

2. configure the job to use ssh

before_script:
    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com".insteadOf "git@gitlab.com:"
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts

The "ssh://git@gitlab.com" bit may be different if you are trying to do git clone git@gitlab.com: or pip install -e git+ssh://git@gitlab.com/...; adjust it accordingly to your needs.

At this point, your CI is able to use ssh to fetch from another (private) repository.

3. [Bonus DRY]

Use this trick to write it generically:

.enable_ssh: &enable_ssh |-
    git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com".insteadOf "ssh://git@gitlab.com"
    mkdir -p ~/.ssh
    chmod 700 ~/.ssh
    echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
    chmod 644 ~/.ssh/known_hosts

and enable it on jobs that need it

test:
    stage: test
    before_script:
        - *enable_ssh
    script:
        - ...

Answer 7

1. If your CI runner is running on a container model, you need to use the deploy key. doc: https://docs.gitlab.com/ee/user/project/deploy_tokens/#git-clone-a-repository

git clone https://<username>:<deploy_token>@gitlab.example.com/tanuki/awesome_project.git

2. Create your deploy token
3. Add your token in CI pipeline Variable
4. make sure your container has the git and change the git URL by insteadOf

  image: docker:latest
  before_script:
    - apk add --no-cache curl jq python3 py3-pip git
    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.come/".insteadOf 'git@gitlab.example.come:'

for replace URL: https://docs.gitlab.com/ee/user/project/working_with_projects.html#authenticate-git-fetches

Answer 8

I had a scenario where I had to use my ssh key in 3 different scripts, so I put the ssh key stuff in a single shell script and called it first, before the other 3 scripts. This ended up not working, I think due to the ssh-agent not persisting between shell scripts, or something to that effect. I ended up actually just outputting the private key into the ~/.ssh/id_rsa file, which will for sure persist to other scripts.

.gitlab-ci.yml

script:
    - ci/init_ssh.sh
    - git push # or whatever you need ssh for

ci/init_ssh.sh

# only run in docker:
[[ ! -e /.dockerenv ]] && exit 0

mkdir -p ~/.ssh
echo "$GITLAB_RUNNER_SSH_KEY" > ~/.ssh/id_rsa
chmod 400 ~/.ssh/id_rsa
echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > /.ssh/config

It works like a charm!

Answer 9

If you are using an alpine-based image (maybe docker:latest or docker:dind), your before_script might look like this:

before_script:
    - apk add --no-cache openssh-client git
    - mkdir -p /.ssh && touch /.ssh/known_hosts
    - ssh-keyscan gitlab.com >> /.ssh/known_hosts
    - echo $SSH_KEY | base64 -d >> /.ssh/id_rsa && chmod 600 /.ssh/id_rsa
    - git clone git@git.myhost.com:mygroup/myproject.git

Answer 10

Adding this to .gitlab-ci.yml did the trick for me. (as mentioned here: https://docs.gitlab.com/ee/user/project/new_ci_build_permissions_model.html#dependent-repositories)

before_script:
    echo -e "machine gitlab.com\nlogin gitlab-ci-token\npassword ${CI_JOB_TOKEN}" > ~/.netrc

(I tried setting up SSH_PRIVATE_KEY as mentioned in one of the answers above, it won't work)

Answer 11

Gitlab 15.9.0 introduces an update to the pre-defined variable CI_JOB_TOKEN. Now you can control other projects' access to your private repository, see the release note and documentation.

Once access is granted, you can clone private repositories by adding this line to your job's scripts or before_scripts.

git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com/<namespace>/<project>

Unfortunately, this still does not play nicely with the submodule integration with Gitlab CI/CD. Instead, I do this in my projects.

# .gitlab-ci.yml
default:
  before_script:
  - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com/<namespace>/<project>.git".insteadOf 'git@gitlab.example.com/<namespace>/<project>.git'
  - git submodule update --init --recursive

or like this

# .gitlab-ci.yml
default:
  before_script:
  - |
    cat << EOF > ~/.gitconfig
    [url "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com/<namespace>/<project>.git"]
        insteadOf = git@gitlab.example.com/<namespace>/<project>.git
    EOF
  - git submodule update --init --recursive

And this is what my .gitmodules would look like

[submodule "my-submodule"]
    path = modules/<project>
    url = git@gitlab.example.com/<namespace>/<project>.git
    branch = main

Hope this help!

Answer 12

Seems there is finally a reasonable solution.

In short as of GitLab 8.12 all you need to do is use relative paths in the .submodules, and the git submodule update --init will simply work