Google App Engine deployment with private git repo in package.json

Question

I have a dependency to a private Bitbucket repo in my package.json

{
   "my-dependency": "git+ssh://git@bitbucket.org/something/my-dependency.git"
}

I followed the instructions given in [1] and [2] and created an SSH key that I encrypted with kms.

I've created a custom cloudbuild.yaml as follows:

# Decrypt the file containing the key
steps:
  - name: 'gcr.io/cloud-builders/gcloud'
    args:
      - kms
      - decrypt
      - --ciphertext-file=bitbucket_rsa.enc
      - --plaintext-file=/root/.ssh/id_rsa
      - --location=global
      - --keyring=default
      - --key=bitbucket-key
    volumes:
      - name: 'ssh'
        path: /root/.ssh

  # Set up git with key and domain
  - name: 'gcr.io/cloud-builders/git'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        chmod 600 /root/.ssh/id_rsa
        cat <<EOF >/root/.ssh/config
        Hostname bitbucket.org
        IdentityFile /root/.ssh/id_rsa
        EOF
        mv known_hosts /root/.ssh/known_hosts
    volumes:
      - name: 'ssh'
        path: /root/.ssh

  # Install
  - name: 'gcr.io/cloud-builders/yarn'
    args: ['install']
    volumes:
      - name: 'ssh'
        path: /root/.ssh

  # Build
  - name: "gcr.io/cloud-builders/yarn"
    args: ["build"]
    volumes:
      - name: 'ssh'
        path: /root/.ssh

  # Deploy
  - name: "gcr.io/cloud-builders/gcloud"
    args: ["app", "deploy", "my-service.yaml"]
    volumes:
      - name: 'ssh'
        path: /root/.ssh

When I run it via gcloud builds submit --config=cloudbuild.yaml steps #0 to #3 run through fine, but step #4 fails because app deploy triggers another yarn install which does not have access to the SSH key defined in steps #0 and #1:

Step #4: INFO     rm_node_modules took 0 seconds
Step #4: INFO     starting: yarn_install
Step #4: INFO     yarn_install yarn install
Step #4: INFO     `yarn_install` stdout:
Step #4: yarn install v1.9.4
Step #4: [1/5] Validating package.json...
Step #4: [2/5] Resolving packages...
Step #4: [3/5] Fetching packages...
Step #4: info Visit https://yarnpkg.com/en/docs/cli/install for     documentation about this command.
Step #4:
Step #4: INFO     `yarn_install` had stderr output:
Step #4: error Command failed.
Step #4: Exit code: 128
Step #4: Command: git
Step #4: Arguments: ls-remote --tags --heads     ssh://git@bitbucket.org/something/my-dependency.git
Step #4: Directory: /workspace
Step #4: Output:
Step #4: Host key verification failed.
Step #4: fatal: Could not read from remote repository.
Step #4:
Step #4: Please make sure you have the correct access rights
Step #4: and the repository exists.
Step #4:
Step #4: ERROR    error: `yarn_install` returned code: 1
Step #4: INFO     yarn_install took 11 seconds
Step #4: INFO     build process for FTL image took 11 seconds
Step #4: INFO     full build took 11 seconds
Step #4: ERROR    `yarn_install` had stderr output:
Step #4: error Command failed.

Thank you for your help!

References:

[1] https://cloud.google.com/cloud-build/docs/access-private-github-repos

[2] Link private repository in packages.json in app deployed to gcloud

Forgot to mention that adding `!node_modules/` in `.gcloudignore`, thus sending all modules to the app deployment yields in a `INVALID_ARGUMENT: This deployment has too many files. New versions are limited to 10000 files for this app` — Treecj, Sep 30 '19 at 15:47

Treecj · Answer 1 · 2019-10-01T07:19:18.827

So apparently it is not possible to provide an SSH key for the gcloud app deploy step. Thus using

{
   "my-dependency": "git+ssh://git@bitbucket.org/something/my-dependency.git"
}

will not work!

The workaround (as mentioned by @JKleinne in the linked thread) is to clone the repo and install it from a local folder:

{
  "my-dependency": "lib/my-dependency"
}

I wrote a small bash script that checks if the repo can be accessed and clones/pulls if it can:

GIT_PROJECT=$1
GIT_REPO=$2
NAME=${GIT_REPO}
REMOTE="git@bitbucket.org:${GIT_PROJECT}/${GIT_REPO}.git"

if [[ ! -d ./lib ]]
then
    mkdir -p ./lib
fi

## Test if git repo is accessible
if ! git ls-remote --exit-code -h ${REMOTE}; then
    echo "Unable to access git repo, skipping"
    exit 0
fi

## Clone or pull
if [[ ! -d ./lib/${NAME} ]]
then
    git clone ${REMOTE} ./lib/${NAME}
else
    git -C ./lib/${NAME} pull
fi

I then use it in the preinstall script:

"preinstall": "./get-internal-package.sh something my-dependency",

score 1 · Answer 2 · answered Sep 30 '19 at 16:05

Whenever there's a Host Key verification failed error, it is most likely related to your known_hosts file, it means that the host key of bitbucket.org is not in your known_hosts file and so the client does not have a way to verify it. Try running ssh-keyscan -t rsa bitbucket.org > known_hosts and then cat known_hosts and see if the bitbucket.org host key is present.

If the answer is no and the output is empty, it could be a network problem interfering with the process. Follow the accepted answer in this thread to troubleshoot it.

If the answer is yes, perfect there is only one step missing and that is to configure your SSH key into Bitbucket. Go into bitbucket.org under "Bitbucket Settings" and then click "SSH Keys" in one of the options. Add a key (label it whichever name you decide) and paste the output you get from running cat ~/.ssh/id_rsa.pub into the key section.

Essentially, in one of the threads you linked I'm cloning the private repository in the same folder as the root of my application. That way, in the package.json I can simply add this line dependencies: {“circular-structure-stringify”: “./circular-structure-stringify”} which will allow me to require() it like any npm package.

I realize I should've kept the Github link open but nevertheless, I am working on another example which I will be posting in the comments section soon. Meanwhile, let me know on the outcome.

Thank you very much for your support @JKleinne. As mentioned in my initial post the `yarn install` from step #2 (count is starting with 0) works fine. In this `yarn install` the SSH Key is available. I also verified this with the `ssh-keyscan` step you suggested. The problem is when running `gcloud app deploy`. — Treecj, Sep 30 '19 at 16:25
I would want to avoid using `dependencies: {“my-dependency”: “./my-dependency”}` since this would require some additional script when calling `yarn install` on the local machine, and for that reason does not scale nicely when adding more dependencies via `git+ssh`. — Treecj, Sep 30 '19 at 16:28
Hey no worries, it's actually the method I used as a workaround to that same issue you are having. It seemed to work perfectly fine when cloning but specifying it in the dependencies as `git+ssh` will give `Host key verification error` which is why I settled with the method above. In any case, I suggest to also create a post in Gitlab's forums. There is surely a way to solve this without having to perform a frankenstein workaround and the guys there will surely be able to provide you with the support needed. I'll still actively try finding a solution to this and will let you know of my finding — JKleinne, Sep 30 '19 at 17:13

Google App Engine deployment with private git repo in package.json

2 Answers2

Linked