1

I am trying to pull large files into a Docker container from git using git-lfs. Unfortunately, I keep getting the error:

...

 ---> f07e7087dc5a
Step 13/16 : RUN git lfs pull
 ---> Running in a387e389eebd
batch response: Git credentials for https://github.XXXX.edu/XXXXX/XXXXXXXXX.git not found.
error: failed to fetch some objects from 'https://github.XXXX.edu/XXXXX/XXXXXXXXX.git/info/lfs'
The command '/bin/sh -c git lfs pull' returned a non-zero code: 2

Any idea how to fix this and get my files pulled correctly and error-free? I am running the following in Docker to try to get git-lfs to work:

# Get git-lfs and pull down the large files
RUN apt-get update && apt-get install -y apt-utils && apt-get install -y curl
RUN curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
RUN apt-get install -y git-lfs
RUN git lfs install
RUN git lfs pull

I add my .gitattributes file and .git file to the Docker image as well.

EDIT: Can I maybe somehow use:

https://you:password@github.com/you/example.git

or

git config remote.origin.url https://you:password@github.com/you/example.git
peachykeen
  • 4,143
  • 4
  • 30
  • 49
  • Your edit is on the right track. You can try `RUN git remote add origin https://you:password@github.com/you/example.git` before your `pull` line. If it fails because `origin` already exists, then you should modify the `.git` file you're copying into the container. – Keenan Lawrence Jun 25 '18 at 23:36

1 Answers1

1

May be I can use https://you:password@github.com/you/example.git:

That is a bad practice, as anyone doing a docker image history on your built image would get those credentials back.

It is better to do a multi-stage build, as described in "Access Private Repositories from Your Dockerfile Without Leaving Behind Your SSH Keys".

It uses an SSH key instead of username/password because:

  • you can generate and register an SSH key dedicated for your docker build.
  • you can revoke that key at any time, since it is used only for this docker build (as opposed to a credential password you cannot easily change without impacting possibly other scripts using said password)

Your Dockerfile would look like:

# this is our first build stage, it will not persist in the final image
FROM ubuntu as intermediate

# install git
RUN apt-get update
RUN apt-get install -y git

# add credentials on build
ARG SSH_PRIVATE_KEY
RUN mkdir /root/.ssh/
RUN echo "${SSH_PRIVATE_KEY}" > /root/.ssh/id_rsa

# make sure your domain is accepted
RUN touch /root/.ssh/known_hosts
RUN ssh-keyscan bitbucket.org >> /root/.ssh/known_hosts

RUN git clone git@bitbucket.org:your-user/your-repo.git

FROM ubuntu
# copy the repository form the previous image
COPY --from=intermediate /your-repo /srv/your-repo
# ... actually use the repo :)

Warning, March 2023, regarding github.com SSH access:

"GitHub has updated its RSA SSH host key"

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • So I changed a few things. Namely, `RUN ssh-keyscan github.mit.edu >> /root/.ssh/known_hosts` and then also `RUN git clone https://github.mit.edu/user/users-repo.git`. But this is a private repository and it is through the enterprise version of github, so when I run things, I get the error: `Cloning into 'users-repo'... fatal: could not read Username for 'https://github.mit.edu': No such device or address` – peachykeen Jun 26 '18 at 14:19
  • 1
    @agaidis Yes. That is why I suggest creating an SSH key dedicated for that clone. Using an *SSH* URL: `RUN gt clone git@github.mit.edu/user/users-repo.git` – VonC Jun 26 '18 at 14:24
  • Now I get: `Cloning into 'users-repo'... Warning: Permanently added the ECDSA host key for IP address 'xx.x.xx.xx' to the list of known hosts. @ WARNING: UNPROTECTED PRIVATE KEY FILE! @ Permissions 0644 for '/root/.ssh/id_rsa' are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored. Load key "/root/.ssh/id_rsa": bad permissions git@github.mit.edu: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.` – peachykeen Jun 26 '18 at 14:39
  • Would I want to do something like `RUN chmod 400 ~/.ssh/id_rsa` as seen in this stackoverflow post? https://stackoverflow.com/questions/29933918/ssh-key-permissions-0644-for-id-rsa-pub-are-too-open-on-mac – peachykeen Jun 26 '18 at 15:00
  • 1
    @agaidis yes indeed: the permissions need to be strict enough for SSH to work. I usually use https://stackoverflow.com/a/37626619/6309 – VonC Jun 26 '18 at 15:02
  • I tried that and now I am getting: `Cloning into 'users-repo'... Warning: Permanently added the ECDSA host key for IP address 'xx.x.xx.xx' to the list of known hosts. Load key "/root/.ssh/id_rsa": invalid format git@github.mit.edu: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.` Any thoughts?? – peachykeen Jun 26 '18 at 15:11
  • @agaidis Yes: docker or not, you need to make sure the public key (\m/id_rsa.pub`) has been published in your private GitHub, for the user 'user' in his/her SSH keys page. – VonC Jun 26 '18 at 15:13
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/173833/discussion-between-agaidis-and-vonc). – peachykeen Jun 26 '18 at 16:18