2

I'm trying to pip install (via a conda env create command) a custom python package from BitBucket during a docker build command without entering the SSH password/passphrase. This question is similar to this other question, but different because the error is occurring during the docker build command.

The environment.yml file for the conda env create command (during the docker build) looks something like this:

name: my_app
channels:
  - defaults
dependencies:
  - pip=21.2.2
  - python=3.8.11
  - pip:
    - git+ssh://git@bitbucket.org/my_org/my_package_repo.git
    - pandas==1.2.5
    - python-dotenv==0.19.0
    - xlrd==2.0.1
prefix: /usr/local/anaconda3/envs/my_app

When the docker build is trying to build the conda environment inside the Docker image, I get this error:

Installing pip dependencies: ...working... Pip subprocess error:
  Running command git clone -q 'ssh://****@bitbucket.org/my_org/my_package_repo.git' /tmp/pip-req-build-3t5mkmnw
  Host key verification failed.
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.
WARNING: Discarding git+ssh://****@bitbucket.org/my_org/my_package_repo.git. Command errored out with exit status 128: git clone -q 'ssh://****@bitbucket.org/my_org/my_package_repo.git' /tmp/pip-req-build-3t5mkmnw Check the logs for full command output.
ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@bitbucket.org/my_org/my_package_repo.git' /tmp/pip-req-build-3t5mkmnw Check the logs for full command output.

Ran pip subprocess with arguments:
['/opt/conda/envs/work_content/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/condaenv.9p_rq9_h.requirements.txt']
Pip subprocess output:
Collecting git+ssh://****@bitbucket.org/my_org/my_package_repo.git (from -r /tmp/condaenv.9p_rq9_h.requirements.txt (line 3))
  Cloning ssh://****@bitbucket.org/my_org/my_package_repo.git to ./pip-req-build-3t5mkmnw

failed

CondaEnvException: Pip failed

This remote/custom package installs successfully when I'm in my terminal and follow the answer in the referenced StackOverflow question. However, when I try to same thing before running the docker build command, I get the error above. I'm assuming that's because Docker is building a whole new OS image, and it no longer has the SSH RSA passphrase I provided in the terminal. How can I get past this error without providing the passphrase during the build?

Update

Based on suggestions in one of the current answers, I've modified my Dockerfile to look something like this:

# syntax=docker/dockerfile:experimental
FROM continuumio/miniconda3:4.10.3
...
RUN apt-get install -y openssh-client git
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts
RUN --mount=type=ssh git clone git@bitbucket.org:my_org/my_package_repo.git /tmp/my_package_repo

The --mount command creates a local copy of the repository in the /tmp directory. So now the conda environment.yml file needs to be changed to pip install from that local directory like this:

name: my_app
channels:
  - defaults
dependencies:
  - pip=21.2.2
  - python=3.8.11
  - pip:
    - /tmp/my_package_repo
    - pandas==1.2.5
    - python-dotenv==0.19.0
    - xlrd==2.0.1
prefix: /usr/local/anaconda3/envs/my_app

... and I'm running the docker build process with commands like these:

eval $(ssh-agent); ssh-add ~/.ssh/id_rsa
DOCKER_BUILDKIT=1 docker build --ssh default .

The eval command above is to manually input my SSH passphrase before the Docker build begins. The DOCKER_BUILDKIT=1 in front of the docker build command forces Docker to build with Docker Buildkit which is required for the RUN --mount=type=ssh git clone command in the Dockerfile. This solution works for me now. It's not exactly what's in the answer below so I thought I'd share this with the community. I'll be marking the answer that pointed me in this direction as the correct answer.

Jed
  • 1,823
  • 4
  • 20
  • 52
  • You can use a `.netrc` file to autenticate inside your docker container https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html – Gonzalo Odiard Nov 29 '21 at 19:14
  • The `.netrc` file seems promising. How exactly should I set this up to work with Docker and bitbucket? – Jed Nov 29 '21 at 19:54
  • use a `.netrc` is safer than using ssh authentication because you can create token with specific uses granted. You can check https://support.atlassian.com/bitbucket-cloud/docs/use-oauth-on-bitbucket-cloud/ but github and gitlab also allow create tokens. Then you create your `.netrc` with your token information and connect to your repo by https:/ not using ssh – Gonzalo Odiard Nov 30 '21 at 13:57

1 Answers1

2

SSH needs access to your private RSA key for the authentication. Your private key is not shared with any command which runs inside a Docker image. Docker has a builtin feature to solve this issue, which is documented here.

Essentially what you want to do is to download your PUBLIC key to the Docker image:

# Download public key for github.com
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts

Define a mount with type ssh in order for Docker Engine to forward SSH agent connections.

RUN --mount=type=ssh git clone git@github.com:myorg/myproject.git myproject

This run command can be something else, for example something which installs your Pythin packages.

You build your docker image with -ssh option:

docker build --ssh default .
Ervin Szilagyi
  • 14,274
  • 2
  • 25
  • 40
  • This answer is calling out GitHub, does the same apply for BitBucket? – Jed Nov 29 '21 at 20:11
  • Does not really matter if you have GitHub or BitBucket. SSH authentication is the same for both. Also, that line was copied from the Docker documentation. It was just an example, since you don't shave your `Dockerfile`. – Ervin Szilagyi Nov 29 '21 at 20:12
  • I was able to figure out the solution based the hints provided in this solution. See the Update in the question for the final solution based on this answer. – Jed Nov 29 '21 at 21:27