Bluemix Dev Ops: Building a project with private git submodules

Question

I would like to know how to build a project with private git submodules using IBM Bluemix Dev Ops Services.

In my pipelines, I have a 'Build' job with the type 'Shell Script':

#!/bin/bash
git submodule init
git submodule update --recursive

But my submodules include a number of private repositories, and I get:

Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

In my local machine, I am able to run those commands because I have access and I am using my key. What can I do to make it work here? I do not wish to commit my private key into git.

The repo for the app I am deploying is hosted on GitHub. And the private submodules are hosted on BitBucket.

Update

I tried to use my private key in the build console, but it did not work:

echo "... my private key ..." >> ~/.ssh/throwaway_key
chmod 400 ~/.ssh/throwaway_key
ssh-agent bash -c 'ssh-add ~/.ssh/throwaway_key; git submodule update --recursive'

Is it not working because I am inside a docker container? Do I have to update /etc/ssh/ssh_config? I don't have access to this inside the container that this job runs in.

Update 2

I also tried without success:

echo "Host            bitbucket.org
    Hostname        bitbucket.org
    IdentityFile    ~/.ssh/throwaway_key
    IdentitiesOnly yes" >> ~/.ssh/config

Can you not paste your key into the build shell script and have it available there? Otherwise Bluemix DevOps does not provide any support for SSH keys — Ben Rondeau, Feb 12 '16 at 06:49
@BenRondeau I tried something based on your suggestion. Could you see my update and give me any other suggestions? — mc9, Feb 18 '16 at 03:08
@BenRondeau It gave me the same error that I don't have access rights. I tried `ssh-agent ...` solution locally and I can confirm it works. I also tried editing `~/.ssh/config` (Please see my another edit) — mc9, Feb 18 '16 at 03:22
Okay. I am out of my depth at this point. I will see if I can get someone from our DevOps team to chime in. Thanks — Ben Rondeau, Feb 18 '16 at 16:37

score 1 · Answer 1 · answered May 10 '16 at 20:13

I have a similar setup. I define a Checkout job, whose job it is to re-extract the source, explicitly passing a password in the clone URL. Once that's done, the submodule update works fine. This is the script:

#!/bin/bash
git clone --recursive https://myname:$PASSWORD@hub.jazz.net/git/myname/my-project
cd my-project
git submodule update --remote

PASSWORD is defined as a Secure property on the Environment Properties tab. It's a bit clunky and non-DRY, but it enabled the behaviour I wanted.

I use the Checkout job as an input to the Build job (I probably could have done it as one big job, but I wanted to be able to visually distinguish failures in checkout and build.)

xverges · Answer 2 · 2019-02-16T08:21:42.973

This worked for me, using a github deploy key for the repo where my submodule resides

    #!/bin/bash

    # Build the private key file from the secret environment setting $id_rsa_abacus_pylib
    echo "-----BEGIN OPENSSH PRIVATE KEY-----" > /home/pipeline/.ssh/id_rsa_abacus_pylib
    (echo $id_rsa_abacus_pylib | tr ' ' '\n' | grep -v -e "----" | grep -v -e OPENSSH | grep -v -e PRIVATE) >> /home/pipeline/.ssh/id_rsa_abacus_pylib
    echo "-----END OPENSSH PRIVATE KEY-----" >> /home/pipeline/.ssh/id_rsa_abacus_pylib
    chmod 400 /home/pipeline/.ssh/id_rsa_abacus_pylib
    #cat /home/pipeline/.ssh/id_rsa_abacus_pylib

    # Replace the SSH command used by git
    echo 'ssh -vvv -i ~/.ssh/id_rsa_abacus_pylib -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no $*' > ssh
    chmod +x ssh

    GIT_SSH="./ssh" git submodule init
    GIT_SSH="./ssh" git submodule update --recursive

In my initial attempts, the operation did not work because I was failing to properly move the SSH key from an environment variable to a file.

It takes advantage from option 3 in https://superuser.com/questions/232373/how-to-tell-git-which-private-key-to-use/868699#868699

score 0 · Answer 3 · answered Oct 06 '20 at 08:43

Here is a simple solution:

Change the Build stage type from "Simple" to "Shell Script".

Add this to script:

 git submodule init
 git config submodule.foo.url https://$USERNAME:$PASSWORD@github.com/foo.git
 git submodule update --recursive

Add USERNAME and PASSWORD in the Environment Properties.

credit: https://stackoverflow.com/a/7714592/1335313

Problem with the above approach is that you need to work in the delivery pipeline to setup for each submodule. It can create a problem when the submodules changed.

Another potential solution is to setup git credential store in the pipeline:

git config credential.helper store
echo "https://LOGIN:${PAT}@github.com/path/to/submodule.git" > ~/.git-credentials

Credit: https://stackoverflow.com/a/63939958/1335313

Read more: https://www.shellhacks.com/git-config-username-password-store-credentials/

Bluemix Dev Ops: Building a project with private git submodules

3 Answers3