21

I have a private repository on a GitLab server and using the SSH I can pull a project using git clone.

But I want to run a script on linux command line directly from the server (more specific, a Drupal / Drush .make file)

I tried to run it using the raw file:

drush make http://server.com/user/project/raw/master/file.make

(for the convenience of non Drupal users let’s say)

curl http://server.com/user/project/raw/master/file.make

Without success. Of course, it returns me the login page.

Is it possible?

tvl
  • 3,868
  • 2
  • 16
  • 35
  • Does `http://user:password@server.com/user/project/raw/master/file.make`, replacing `user` and `password` with your credentials, work? – ChrisGPT was on strike Jun 13 '14 at 14:40
  • Thank for the help but it doesn't work. Also I would like a solution without exposing my password. (but even if that works it will be really good!) – tvl Jun 13 '14 at 14:58
  • 1
    You're going to have a tough time accessing files over HTTP without exposing your password. In theory you could use a *client-side* SSL certificate to authenticate, but that's likely a lot of work. I seriously doubt that it's supported by GitLab. You may want to create a dedicated read-only account and use that in your command to limit your exposure. – ChrisGPT was on strike Jun 13 '14 at 15:07
  • 1
    http://www.systemseed.com/blog/drush-make-private-git-repository-github – ChrisGPT was on strike Jun 13 '14 at 15:11
  • To be honest my final goal is to grab the .make files from a GitLab server and feed them on a Aegir server so it can create "platforms". But, for the moment, I try the middle linux-steps (as Aegir uses drush commands). So if I add on the Aegir server an SSL certificate I can verify it on the GitLab and will have a password-less client-to-server communication? – tvl Jun 13 '14 at 15:23
  • Only if you manage to get GitLab to rely on client SSL certificates as an authentication method. If GitLab offers something similar to GitHub's API, the link I added above may get you started. – ChrisGPT was on strike Jun 13 '14 at 15:32
  • 1
    Man you are awesome! I knew about the tokens, but I didn't try them. Thanks for the guid-inspiration. GitLab have tokens and an excellent API. [here](https://github.com/gitlabhq/gitlabhq/blob/master/doc/api/README.md#gitlab-api). So if you want to grab a raw file you must do somethink like: http://gitlabserver.com/api/v3/projects/:project-id/repository/raw_blobs/:sha?private_token=:token. – tvl Jun 13 '14 at 16:26
  • Glad to see that this helped. Once you've got it working, please consider answering your own question with details about what you needed to do. I don't have a GitLab box handy and don't feel comfortable writing an answer without one. – ChrisGPT was on strike Jun 13 '14 at 16:29
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/55585/discussion-between-tolis-and-chris). – tvl Jun 13 '14 at 16:30

3 Answers3

28

With Chris's valuable help, here is how you can run a script (drupal .make file in my case) from a GitLab server. (Probably it works for GitHub but I didn't test it. Maybe the syntax will be a bit different). (Of course this works for any type of script)

It can be done using the authentication tokens. Here is the documentation of the GitLab's API and here is the GitHub's API

For convenient I will use the https://gitlab.com as the example server.

  • Go to https://gitlab.com/profile/account and find your "Private token"

  • Then print the list of the projects and find the id of your project you are looking for

    curl https://gitlab.com/api/v3/projects?private_token=<your_private_token>

    or go there with your browser (a json viewer will help a lot)

  • Then print the list of the files that are on this project and find the id of your file you are looking for

    curl https://gitlab.com/api/v3/projects/<project_id>/repository/tree?private_token=<your_private_token>

  • Finally get / run the file!

    curl https://gitlab.com/api/v3/projects/<project_id>/repository/raw_blobs/<file_id>?private_token=<your_private_token>

In case you want to run the script (drupal .make)

drush make https://gitlab.com/api/v3/projects/<project_id>/repository/raw_blobs/<file_id>?private_token=<your_private_token> <drupal_folder>

(If you are here looking for a workflow to integrate GitLab with Aegir .make platforms without using tokens (maybe SSH?) please make a thread and paste here the link.)

EDIT

You can get the file without the project_id by using the encoded project name. For example the my-user-name/my-project will become: my-user-name%2Fmy-project

tvl
  • 3,868
  • 2
  • 16
  • 35
  • How to access a file in a folder? Is it possible to use a fixed file id? The id changes every time I update the file. – Bruce Yong Li Oct 29 '15 at 21:24
  • Sorry @Yong I can't help you as I don't use this method for long time now because it wasn't practical. – tvl Oct 30 '15 at 14:55
  • 10
    Thanks for your reply. I figured out another way to make it work. This is how I did: `https://gitlab.com///raw/master//?private_token=` – Bruce Yong Li Oct 30 '15 at 22:36
  • @Yong doesn't work for me (or rather, only works as long as i also use a cookie session which is already logged in, and the private_token is ignored – hanshenrik Jan 03 '18 at 19:19
  • Currently the private token is probably called 'access token'. I got one from https://gitlab.com/profile/personal_access_tokens and have used it successfully for downloading from a private repository with wget (https://stackoverflow.com/a/31442131/5459638 for the interested ones). By asking for an access token you will need to specify expiry date and purpose (in my case, read_repository) – XavierStuvw Nov 22 '18 at 14:19
  • I downvoted the answer and @BruceYongLi's because the path to access a file in a folder does not work. I found GitLab very hostile than GitHub, regards cURL, SVN and wget. I'll switch to GitHub and use the old and good SVN. – Oo'- Jun 01 '20 at 01:41
11

Update 2018-12-25:

as long as you're not downloading huge files, this should work:

curl -s -H "Private-Token: <token>" "https://gitlab.com/api/v4/projects/<urlencode("gitlab_username/project_name")>/repository/files/<path/to/file>/raw?ref=<branch_name>"

, a real example, downloading the file /README.md from the private repository https://gitlab.com/divinity76/Yur17, where the web download url is https://gitlab.com/divinity76/Yur17/raw/master/README.md?inline=false, is:

curl -s -H "Private-Token: afF2s1xgk6xcwXHy3J4C" "https://gitlab.com/api/v4/projects/divinity76%2Fyur17/repository/files/README%2Emd/raw?ref=master"

take special note of how the gitlab_username/repo_name was url-encoded, eg / became %2F (you can check how your username & repo name is as url-encoded by opening your browser javascript terminal and write encodeURIComponent("your_username/repo_name"); in the terminal and press enter.)

thanks to Jonathan Hall @ gitlab at mg.gitlab.com, and https://docs.gitlab.com/ee/api/repository_files.html , and tvl for helping reach a solution.

Update 2018-12-11: this method no longer works, now it just serves the login page, with a message saying need to log in to continue, even using HTTP 200 OK (shame on them), will update if i find out why ( @XavierStuvw claims it's security concerns related)

i found a much easier way to do it than @tvl 's answer,

first create an access token with API access here: https://gitlab.com/profile/personal_access_tokens , then do:

wget --header 'PRIVATE-TOKEN: <token>' 'https://gitlab.com/<username>/<repo>/raw/master/path/to/file.ext'

i found the solution here.

oliolioli
  • 91
  • 2
  • 5
hanshenrik
  • 19,904
  • 4
  • 43
  • 89
  • This solution has also worked out in https://stackoverflow.com/a/31442131/5459638, and for me – XavierStuvw Nov 22 '18 at 14:36
  • This no longer works for security concerns - but the API is available and you can just use the `read_repository` scope (no need for the API scope). – BLuEGoD Dec 11 '18 at 10:40
  • @BLuEGoD yeah doesn't work now :< - werid, TLS certificates are verified before headers are sent so it's probably not a MITM concern, do you know what security concerns they are talking about? – hanshenrik Dec 11 '18 at 12:02
  • @hanshenrik see "Improper Enforcement of Token Scope" in https://about.gitlab.com/2018/11/28/security-release-gitlab-11-dot-5-dot-1-released/ – BLuEGoD Dec 11 '18 at 15:14
  • @BLuEGoD thanks for the heads-up, found a new (albeit terrible from a bandwidth point-of-view) way to do it, that is still easier than tvl's answer ^^ – hanshenrik Dec 25 '18 at 13:10
  • 1
    @hanshenrik Regarding your update "Update 2018-12-25" with gitlab-API v4: please add a "/raw" before ?ref=" then you can remove the | jd ... > filename.ext – oliolioli Jan 17 '19 at 12:56
  • For gitlab-API v4 I had to insert instead of – oliolioli Jan 17 '19 at 12:58
  • 1
    must also be urlencoded, e.g. . with %2E and / with %2F. – oliolioli Jan 17 '19 at 13:14
3

If you would like to access a file from private GitLab, you could use the below approach which worked for me:)

Construct the URL:

https://url/api/v4/projects/projectId/repository/files/fileName/raw?ref=master&private_token=Generated_private_token

  • url is your Gitlab url ex: git.lab.com.
  • /api/v4/projects is a constant.
  • projectId is the projectId of your project, which you can find below the name of your project in gitlab.
  • /repository/files is again a constant.
  • fileName is the name of the file ex: sagar.txt
  • /raw?ref= is a constant and the value of ref can be master or any branch which you would like to take the file from. I am retrieving the file from Master.
  • Generated_private_token should be generated from gitlab, please follow the steps in mentioned in the link : Generate Private Token
Sagar Shetty
  • 165
  • 2
  • 5
  • Note that file path needs to be url encoded ie. `app/models/key.rb` goes in the url as `app%2Fmodels%2Fkey%2Erb` – lights Feb 25 '21 at 14:28
  • This is the answer I was looking for. GitLab docs suck if you use private GitLab. This one works great. Thanks! – C. Feng Apr 08 '21 at 20:06
  • This returns `{"message":"404 Project Not Found"}` even though the repo exists with all the necessary perms to the token – Nikhil Nanjappa Jan 18 '23 at 14:35