GitLab CI enable SCP

Question

I'm using one of the shared runners on GitLab.com currently. Is it possible to set up the .gitlab-ci.yaml file such that a build can SCP files from a remote server onto the runner? My goal is to SCP files that are necessary dependencies for my build, but they are not tracked in any Git repositories.

I've marked the line where I would like to be able to perform the transfer, but I don't know how to express it properly.

Note: CodeA has dependencies in CodeB and CodeC, which must be built before CodeA can compile, so I need to have access to CodeB and CodeC to build them first on the ubuntu image.

image: ubuntu:12.04

before_script:

build_CodeC:
  stage: build
  allow_failure: true
  script:
-->- scp user@remoteServer:/home/user/file.tar . <---
   - sh ./continuous_integration/build_CodeC_dependency.sh

build_CodeB:
  stage: build
  script:
    - sh ./continuous_integration/build_CodeB_dependency.sh

build_CodeA:
  stage: build
  script:
    - sh ./continuous_integration/build_CodeA.sh

Answer 1

From your question here, I think that getting your dependencies through http is not possible, so here's what you need to do in order to use scp:

• Generate a key pair
• Copy the private key to a gitlab CI variable (let's call it SSH_PRIVATE_KEY)
• Copy the public key to the server gitlab will connect to and add it to your ~/.ssh/authorized_keys file
• Tell your CI pipeline to use the private key that is stored in the Gitlab CI variable

In order to do that last step, just add the following to your .gitlab-ci.yml in the script or before_script section of the job of interest:

- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# Run ssh-agent (inside the build environment)
- eval $(ssh-agent -s)
# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

You may also want to specify that CodeA is dependent on B and C. In order for that to work, build_CodeB and build_CodeC need to be in a different stage than build_CodeA.

Besides that, you need a way to carry the built files from build_CodeB and build_CodeC jobs to the build_CodeA job. One way to do that is to use artifacts.

In the end, your .gitlab-ci.yml file should look something like this:

image: ubuntu:12.04

stages:
  - deps
  - build

build_CodeC:
  stage: deps
  allow_failure: true
  script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    # Run ssh-agent (inside the build environment)
    - eval $(ssh-agent -s)
    # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
    - ssh-add <(echo "$SSH_PRIVATE_KEY")
    - mkdir -p ~/.ssh
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
    - scp user@remoteServer:/home/user/file.tar .
    - sh ./continuous_integration/build_CodeC_dependency.sh
  artifacts:
    paths:
      - path_to_built_codeC

build_CodeB:
  stage: deps
  script:
    - sh ./continuous_integration/build_CodeB_dependency.sh
  artifacts:
    paths:
      - path_to_built_codeB

build_CodeA:
  stage: build
  dependencies:
    - build_CodeB
    - build_CodeC
  script:
    - sh ./continuous_integration/build_CodeA.sh

I only put the SSH key setup part in build_CodeC because that's where you are using scp. You would need to copy this over to any job that would require to use scp. I'm thinking you may need to do this in build_codeB since your tar file will not be carried to the build_CodeB job.