7

I'm trying to build a Docker container that should install a series of python packages from a requirements.txt file. One of the entries is a python package hosted on a private GitHub repository. To install it, I've created a pair of SSH keys and added the public one as a Deploy Key to the GitHub repository.

However, when I'm building the container I'm getting this error:

ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@github.com:organization/my-package' /tmp/pip-install-e81w4wri/my-package Check the logs for full command output.

I've tried to debug the error by changing the pip install command of the docker file with RUN git clone git@github.com:organization/my-package.git and it did work fine.

What does this error mean and how can I solve it? I could clone it and install it with a dedicated command, but if possible I'd like to keep all requirements in a single place. Thanks!

This is the Dockerfile that I'm using:

FROM joyzoursky/python-chromedriver:3.7-alpine3.8 as base

FROM base as builder

RUN echo "http://dl-8.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
RUN apk --no-cache --update-cache add bash gcc gfortran build-base git wget freetype-dev libpng-dev openblas-dev openssh-client
RUN ln -s /usr/include/locale.h /usr/include/xlocale.h

# copy requirements
RUN mkdir /install
WORKDIR /install
COPY ./requirements.txt /var/www/requirements.txt

### GITHUB SSH KEY ###
COPY ./keys/deploy_key_private .
RUN mkdir /root/.ssh && mv deploy_key_private /root/.ssh/id_rsa
RUN eval $(ssh-agent) && \
    ssh-add /root/.ssh/id_rsa && \
    ssh-keyscan -H github.com >> /etc/ssh/ssh_known_hosts

RUN pip install --upgrade pip && pip install --prefix=/install -r /var/www/requirements.txt --log logs.txt

FROM base
COPY --from=builder /install /usr/local
# KEEP ON BUILDING THE CONTAINER

The package is listed in the requirements.txt as git+ssh://git@github.com:organization/my-package@master#egg=my_package

If relevant, here is the traceback from pip:

Exception information:
2020-07-30T11:56:55,329 Traceback (most recent call last):
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
2020-07-30T11:56:55,329     status = self.run(options, args)
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
2020-07-30T11:56:55,329     return func(self, options, args)
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 325, in run
2020-07-30T11:56:55,329     reqs, check_supported_wheels=not options.target_dir
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
2020-07-30T11:56:55,329     discovered_reqs.extend(self._resolve_one(requirement_set, req))
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
2020-07-30T11:56:55,329     abstract_dist = self._get_abstract_dist_for(req_to_install)
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 340, in _get_abstract_dist_for
2020-07-30T11:56:55,329     abstract_dist = self.preparer.prepare_linked_requirement(req)
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 469, in prepare_linked_requirement
2020-07-30T11:56:55,329     hashes=self._get_linked_req_hashes(req)
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 239, in unpack_url
2020-07-30T11:56:55,329     unpack_vcs_link(link, location)
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 99, in unpack_vcs_link
2020-07-30T11:56:55,329     vcs_backend.unpack(location, url=hide_url(link.url))
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/vcs/versioncontrol.py", line 733, in unpack
2020-07-30T11:56:55,329     self.obtain(location, url=url)
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/vcs/versioncontrol.py", line 641, in obtain
2020-07-30T11:56:55,329     self.fetch_new(dest, url, rev_options)
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/vcs/git.py", line 230, in fetch_new
2020-07-30T11:56:55,329     self.run_command(make_command('clone', '-q', url, dest))
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/vcs/versioncontrol.py", line 774, in run_command
2020-07-30T11:56:55,329     log_failed_cmd=log_failed_cmd)
2020-07-30T11:56:55,329   File "/usr/local/lib/python3.7/site-packages/pip/_internal/vcs/versioncontrol.py", line 166, in call_subprocess
2020-07-30T11:56:55,329     raise SubProcessError(exc_msg)
2020-07-30T11:56:55,329 pip._internal.exceptions.SubProcessError: Command errored out with exit status 128: git clone -q 'ssh://****@github.com:organization/my-package' /tmp/pip-install-e81w4wri/my-package Check the logs for full command output.
arabinelli
  • 1,006
  • 1
  • 8
  • 19
  • I'm not too familiar with Alpine, but usually the keys for known hosts don't go into `/etc/ssh/ssh_known_hosts`, but `~/.ssh/known_hosts`. – Marko E Jul 30 '20 at 14:09
  • Thanks for your comment, I tried to change the path but unfortunately it didn't solve the issue. I don't necessarily have to stick with Alpine, do you think the issue might be solved with a different base image? – arabinelli Jul 30 '20 at 14:17
  • Not sure, I don't think so. Does this post answer help: https://stackoverflow.com/questions/9617336/how-to-resolve-git-did-not-exit-cleanly-exit-code-128-error-on-tortoisegit? In short, try adding git configuration with user name and user email. – Marko E Jul 30 '20 at 15:05

1 Answers1

6

git@github.com:organization/my-package.git is a valid SSH URL. ssh://git@github.com:organization/my-package.git is not. ssh://git@github.com/organization/my-package.git would be.

As in here, you can add GIT_SSH_COMMAND='ssh -v' pip install ... to see exactly what is going on.

You might need:

git config --global url."ssh://git@github.com/".insteadOf ssh://git@github.com:

The OP arabinelli reports in the comments having to use the following line in requirements.txt:

git+ssh://git@github.com/my-organization/my-repo-name@master#egg=my_package_dir

Aug. 2022, Jako adds in the comments:

This worked for me with a private BitBucket repository:

git+ssh://git@bitbucket.org/my-organization/my-repo-name@master#egg=my_project&subdirectory=subdir1

                                      ^^^^^^^

I had to specify the subdirectory 'subdir1'

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • It was an URL issue indeed, changing it in the requirements.txt solved it. Thanks!! – arabinelli Jul 31 '20 at 10:59
  • @arabinelli could you tell me exactly how the url was structured in the requirements.txt file? – Alex F Oct 07 '20 at 14:38
  • This is the line as it appears on the requirements.txt (I had to replace sensitive info as it's a private package): `git+ssh://git@github.com/my-organization/my-repo-name@master#egg=my_package_dir` – arabinelli Oct 11 '20 at 13:25
  • 1
    @arabinelli Thank you for this feedback. I have included your comment in the answer for more visibility. – VonC Oct 11 '20 at 13:27
  • This worked for me with a private bitbucket repo: `git+ssh://git@bitbucket.org/my-organization/my-repo-name@master#egg=my_project&subdirectory=subdir1` I had to specify the subdirectory 'subdir1' – Jako Aug 21 '22 at 14:19
  • 1
    @Jako Thank you for the feedback. I have included your comment in the answer for more visibility. – VonC Aug 21 '22 at 18:08