Cannot push from gitlab-ci.yml

Question

With my colleagues, we work on a C++ library that becomes more and more important each day. We already built continuous integration utilities through the gitlab-ci.yml file that let us:

Build & Test in Debug mode
Build & Test in Release mode
Perform safety checks like memory leaks using Valgrind and checking if there is any clear symbol in our library we don't want inside it
Generate documentation

All kind of stuff that made us choose GitLab !

We would like to profile our whole library and push the benchmarks in a separate project. We have already done something like for out documentation using the SSH key method but we would like to avoid this this time.

We tried a script like this:

test_ci_push:
  tags:
    - linux
    - shell
    - light
  stage: profiling
  allow_failure: false
  only:
    - new-benchmark-stage
  script:
    - git clone http://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.mycompany.home/developers/benchmarks.git &> /dev/null
    - cd benchmarks
    - touch test.dat
    - echo "This is a test" > test.dat
    - git config --global user.name "${GITLAB_USER_NAME}"
    - git config --global user.email "${GITLAB_USER_EMAIL}"
    - git add --all
    - git commit -m "GitLab Runner Push"
    - git push http://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.mycompany.home/developers/benchmarks.git HEAD:master
    - cd ..

We also tried a basic git push origin master to push our updated files but each time we got the same answer:

remote: You are not allowed to upload code for this project.
fatal: unable to access 'http://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.mycompany.home/developers/benchmarks.git/': The requested URL returned error: 403

Both projects are under the same site and I have the rights to push in both. Where am I doing anything wrong here ?

Clive Makamara · Accepted Answer · 2019-04-25T06:31:38.717

68

The gitlab ci token is more like the deploy key in github.com, so it only has read access to the repository. To actually push you will need to generate a personal access token and use that instead.

First you need to generate the token as shown here in the gitlab documentation. Make sure you check both the read user and api scopes. Also this only works in GitLab 8.15 and above. If you are using an older version and do not wish to upgrade there's an alternative method I could show you but it is more complex and less secure.

In the end your gitlab-ci.yml should look something like this:

test_ci_push:
  tags:
    - linux
    - shell
    - light
  stage: profiling
  allow_failure: false
  only:
    - new-benchmark-stage
  script:
    - git clone http://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.mycompany.home/developers/benchmarks.git &> /dev/null
    - cd benchmarks
    - echo "This is a test" > test.dat
    - git config --global user.name "${GITLAB_USER_NAME}"
    - git config --global user.email "${GITLAB_USER_EMAIL}"
    - git add --all
    - git commit -m "GitLab Runner Push"
    - git push http://${YOUR_USERNAME}:${PERSONAL_ACCESS_TOKEN}@gitlab.mycompany.home/developers/benchmarks.git HEAD:master
    - cd ..

edited Apr 25 '19 at 06:31

answered Sep 29 '17 at 07:53

Clive Makamara

2,845
16
31

1

I want to use my user token **to do a git push while in CI**. I have generated an `API access token`, and I use it in addition of my git https url. That doesn't work, it says that I don't have push permissions while I am the owner. How can I do ? (Without SSH) – Dimitri Kopriwa May 25 '18 at 05:55
Each access token has scopes attached to it, the access token needs the api scope otherwise you can't do anything with it – Clive Makamara May 25 '18 at 06:34
3

I have checked all the scopes, including api scope and I still can't push to my repository. Are you certain that it is possible using a token. I was using ssh manually configured before because I thought it was the one solution – Dimitri Kopriwa May 27 '18 at 13:43
Thankyou! That push step is all I needed to realize how to push my package.json changes for npm versioning! – Jimmy Hoffa Jun 29 '18 at 14:04
Just a minor annotation to your script code: You don't need to `touch test.dat` first. When redirecting stdout to a file using `>`, this file is automatically created. – Mecki Apr 24 '19 at 14:37
Please use the `${CI_REPOSITORY_URL}`. You can also try `${GITLAB_USER_LOGIN}` instead of `${YOUR_USERNAME}` – Melroy van den Berg Dec 14 '21 at 01:04
doesn't work when the gitlab server uses 2FA as is common in enterprise environment. – Eric Jun 01 '23 at 16:37

score 33 · Answer 2 · edited Dec 14 '21 at 00:55

33

While the previous answers are more or less fine, there are some important gotya's.

  before_script:
    - git config --global user.name "${GITLAB_USER_NAME}"
    - git config --global user.email "${GITLAB_USER_EMAIL}"
  script:
    - <do things>
    - git push "https://${GITLAB_USER_LOGIN}:${CI_GIT_TOKEN}@${CI_REPOSITORY_URL#*@}" "HEAD:${CI_COMMIT_TAG}"

For one, we only need to set the username/email to please git.

Secondly having it in the before script, is not super crucial, but allows for easier reuse when doing 'extend'.

Finally, pushing https is 'fine' but since we're not using a stored ssh key, we should avoid anything that can reveal the token. For one, while gitlab won't print the token in this command, git will happily inform us that the new upstream is set to https://username:thetokeninplaintexthere@url So there's your token in plain text, so don't use -u to set an upstream.

Also, it's not needed, we are only doing a single push.

Further more, when determining the URL, I found that using the exist CI_REPOSITORY_URL to be the most reliable solution (when moving repo's for example or whatnot). So we just replace the username/token in the URL string.

edited Dec 14 '21 at 00:55

Melroy van den Berg

2,697
28
31

answered Mar 25 '19 at 19:02

oliver

761
6
4

can you tell us what `@${CI_REPOSITORY_URL#*@}` exactly is? How is the replacement working and where do I find a reference to this syntax? – Fab1n Dec 09 '21 at 16:25
2

The `@` is actually part the HTTP protocol, saying you want to login with username:password@hostname. The variable: `${CI_REPOSITORY_URL}` is the variable. "#*@" is regular expression (regex), but I also don't know what it does. Alternatively, you can also use: `${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git` after the `@`-symbol. – Melroy van den Berg Dec 14 '21 at 01:01
Instead of using the commit tag. I'm actually using `"HEAD:${CI_DEFAULT_BRANCH}"` – Melroy van den Berg Dec 14 '21 at 01:02
This solved my issue but then I went with using a private runner and using ssh key of the runner – mujib ishola Jan 03 '22 at 20:27
4

`#*@` is substring removal, will remove everything from begin to `@` – Miaonster Jun 19 '22 at 09:58
2

I was wondering as well, why this `${CI_REPOSITORY_URL#*@}` was used. The comment from @miaonster put me on the right track: `${CI_REPOSITORY_URL}` expands to something like `https://gitlab-ci-token:[masked]@example.com/gitlab-org/gitlab-foss.git` as you can see from [the gitlab docs](https://docs.gitlab.com/ee/ci/variables/#list-all-environment-variables) – and since we only need everything after the "@", this is done `${CI_REPOSITORY_URL#*@}` and we get `example.com/gitlab-org/gitlab-foss.git`. – jmk Jul 28 '22 at 17:16

score 11 · Answer 3 · edited Dec 14 '21 at 00:56

You could also provide user and password (user with write access) as secret variables and use them.

Example:

before_script:
 - git remote set-url origin https://$GIT_CI_USER:$GIT_CI_PASS@$CI_SERVER_HOST/$CI_PROJECT_PATH.git
 - git config --global user.email 'myuser@mydomain.com'
 - git config --global user.name 'MyUser'

You have to define GIT_CI_USER and GIT_CI_PASS as secret variables (you could always create dedicated user for this purpose).

With this configuration you could normally work with git. I'm using this approach to push the tags after the release (with Axion Release Gradle Pluing - http://axion-release-plugin.readthedocs.io/en/latest/index.html)

Example release job:

release:
  stage: release
  script:
    - git branch
    - gradle release -Prelease.disableChecks -Prelease.pushTagsOnly
    - git push --tags
  only:
   - master

I updated this answer with `$CI_SERVER_HOST` instead of gitlab.com. — Melroy van den Berg, Dec 14 '21 at 01:02

Melroy van den Berg · Answer 4 · 2021-12-14T01:32:20.037

I'm using the following GitLab job:

repo_pull_sync:
  image: danger89/repo_mirror_pull:latest
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
    - if: $REMOTE_URL
    - if: $REMOTE_BRANCH
    - if: $ACCESS_TOKEN
  before_script:
    - git config --global user.name "${GITLAB_USER_NAME}"
    - git config --global user.email "${GITLAB_USER_EMAIL}"
  script:
    - git checkout $CI_DEFAULT_BRANCH
    - git pull
    - git remote remove upstream || true
    - git remote add upstream $REMOTE_URL
    - git fetch upstream
    - git merge upstream/$REMOTE_BRANCH
    - git push "https://${GITLAB_USER_LOGIN}:${ACCESS_TOKEN}@${CI_REPOSITORY_URL#*@}" "HEAD:${CI_DEFAULT_BRANCH}"

I'm using my own danger89/repo_mirror_pull docker image based on alpine, check this GitHub repository for more info.

This GitLab job pull upstream changes from the predefined remote repository + branch (see variables below), and merge them locally in CI/CD and pushes them in GitLab again.

Basically I created a repository pull mirror (which is officially not available for free on GitLab CE, only a push mirror is supported in GitLab).

Create in GitLab a Project Access Token first in GitLab. Via: Settings->Access Tokens. Check 'api' as the scope.
Create a new schedule, via: CI/CD->Schedules->New schedule. With the following 3 variables:
- REMOTE_URL (example: https://github.com/project/repo.git)
- REMOTE_BRANCH (example: master)
- ACCESS_TOKEN: (see Access Token in the first step! Example: gplat-234hcand9q289rba89dghqa892agbd89arg2854, )
Save pipeline schedule

Again, see also: https://github.com/danger89/repo_pull_sync_docker_image

Regarding the question, see the git push command above, which allows you to push changes back into GitLab using GitLab (project) access token.

score 1 · Answer 5 · answered Feb 26 '23 at 15:28

1

Check if GitLab 15.9 (February 2023) can help.

First, the documentation "GitLab CI/CD job token" does mention:

The token has the same permissions to access the API as the user that caused the job to run.

A user can cause a job to run by taking action like pushing a commit, triggering a manual job, or being the owner of a scheduled pipeline.
Therefore, this user must be assigned to a role that has the required privileges.

Second, GitLab 15.9 (for Saas gitlab.com or self-managed instances) proposes:

Control which projects can access your project with a CI/CD job token

The CI/CD job token stored in the CI_JOB_TOKEN CI/CD variable makes authenticating API calls within jobs more intuitive, enabling advanced automation.
For example, the token can be used with bot automation projects that run pipelines in other projects.
The token is short-lived, but has the same permissions as the user that triggered the pipeline.

In an effort to make its usage even more secure we are adding a setting that lets you define a list of trusted projects that you allow to use job tokens to access your project.
This added layer of security means that only these projects will be able to access your project’s API with a job token.
In the bot automation example, it lets you control which bot projects can interact with your own project.

This setting is currently disabled by default in existing projects to avoid impacting pipelines, but we strongly recommend enabling it in all your projects.
It is enabled by default for all new projects.

See Documentation and Issue.

Projects added to the allowlist can make API calls from running pipelines by using the CI/CD job token.

So if you need to push to a separate project, add to its inbound access list your current project. And your pipeline (using your current project) will be able to push to your separate project, using the default CI_JOB_TOKEN token.

answered Feb 26 '23 at 15:28

VonC

1,262,500
529
4,410
5,250

I found this answer doesn't work for me (not sure why). For the same repository, my understanding from this answer is you should be able to commit and push back to the repo in a CI job, but I just get a 403 here: "remote: You are not allowed to upload code". Read-only access seems to be ok, but not write-access. – kratsg Feb 27 '23 at 14:54
@kratsg Do you have a GitLab instance 15.9? And did you add your project to the inbound list of the target project you want to push to? – VonC Feb 27 '23 at 15:01
I have this feature enabled on 15.9 and the target repository is itself (so I didn't need to add it). I did make sure the switch was enabled to limit the feature as described. Do you know how to confirm the permissions of CI_JOB_TOKEN? – kratsg Feb 27 '23 at 16:32
I suspect this is still an open issue given the discussion here (https://gitlab.com/groups/gitlab-org/-/epics/3559#note_846650238). – kratsg Feb 27 '23 at 16:45
@kratsg For testing, try and add the project to itself (to its own inbound list). Regarding permission, try `curl --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/1/members/me"`, which should return your user ID, username, name, state, access level and expires at date for that project (knowing that, to push to a repository, you need to have at least Developer role for that project). – VonC Feb 27 '23 at 19:50
I'm a maintainer on the project. I cannot add the project to itself (it's already on the inbound list and cannot be deleted or added) and the permissions show the access level is right. I still get a 403 unfortunately, so it must be something about write access. I can confirm read-access is fine tho. – kratsg Feb 28 '23 at 16:34
@kratsg OK: asking a separate question might yield more result, especially if you detail the specifics of your current setup. – VonC Feb 28 '23 at 20:15

Cannot push from gitlab-ci.yml

5 Answers5

Control which projects can access your project with a CI/CD job token

Linked