How to prevent Dockerfile caching git clone

Question

I have a Dockerfile trying to package and deploy a web app to a container. The code of app fetches from git repository during Docker image building. Here's the Dockerfile snapshot:

........
RUN git clone --depth=1 git-repository-url $GIT_HOME/
RUN mvn package -Dmaven.test.skip
........

I want the docker do not cache the step of RUN git clone --depth=1 git-repository-url $GIT_HOME/ so that the on-going updated on the the repository can be reflected on the Docker image building. Is it possible to a achieve that?

anq · Answer 1 · 2020-06-25T12:38:30.873

80

Another workaround:

If you use GitHub (or gitlab or bitbucket too most likely) you can ADD the GitHub API's representation of your repo to a dummy location.

ADD https://api.github.com/repos/$USER/$REPO/git/refs/heads/$BRANCH version.json
RUN git clone -b $BRANCH https://github.com/$USER/$REPO.git $GIT_HOME/

The API call will return different results when the head changes, invalidating the docker cache.

If you're dealing with private repos you can use github's x-oauth-basic authentication scheme with a personal access token like so:

ADD https://$ACCESS_TOKEN:x-oauth-basic@api.github.com/repos/$USER/$REPO/git/refs/heads/$BRANCH version.json

(thx @captnolimar for a suggested edit to clarify authentication)

edited Jun 25 '20 at 12:38

answered Sep 01 '16 at 18:07

anq

3,102
21
16

1

Worked perfect for GitHub. Any example for private GitLab? – Ivan Oct 24 '18 at 12:22
2

Nice solution. Heres how to do it for bitbucket. `ADD https://[USER]:[PASS]@api.bitbucket.org/2.0/repositories/[ORG-NAME]/[REPO-NAME]/commit/[BRANCH] /info` `RUN git clone --depth 1 --branch [BRANCH] https://[USER]:[PASS]@bitbucket.org/[ORG-NAME]/[REPO-NAME].git /repo` I recommend creating a specific "Integration" user and using "App passwords" for bitbucket. https://confluence.atlassian.com/bitbucket/app-passwords-828781300.html Combine this with multi-stage build, and it seems like the perfect solution so far. – McP Apr 24 '19 at 10:35
Hi @Ivan look at: https://stackoverflow.com/a/59685604/2992810 – Wazime Jan 10 '20 at 16:37
4

Do do it in a completely generic provider independent way, here's an option `ADD http://worldtimeapi.org/api/ip /time.tmp` – Jay Jan 14 '20 at 03:13
4

As in 2020, the github api appears to have changed and the URL `https://api.github.com/repos/$USER/$REPO/git/refs/heads/$BRANCH` doesn't work anymore, due to api updates? Please suggest the updated URL. – Hemant Kumar Jun 19 '20 at 09:57
@HemantArora Hmm, what response are you getting? It seems to work for me: https://api.github.com/repos/dokku/dokku/git/refs/heads/master Are you certain that the user, repo, and branch are correct and visible to the authenticated user? – anq Jun 25 '20 at 12:37
@HemantArora See https://docs.github.com/en/rest/overview/troubleshooting#basic-authentication-errors `On November 13, 2020 username and password authentication to the REST API and the OAuth Authorizations API were deprecated and no longer work.` The solution given is to use curl to provide the authorization header. – ptrin Jul 12 '21 at 13:44

score 16 · Answer 2 · edited Oct 03 '18 at 10:11

16

Issue 1996 is not yet available, but you have the following workaround:

FROM foo
ARG CACHE_DATE=2016-01-01
RUN git clone ...

docker build --build-arg CACHE_DATE=$(date) ....

That would invalidate cache after the ARG CACHE_DATE line for every build.

Or:

ADD http://www.convert-unix-time.com/api?timestamp=now /tmp/bustcache
RUN git pull

That would also invalidate cache after this ADD line.

Similar idea:

Add ARG command to your Dockerfile:
# Dockerfile
# add this and below command will run without cache
ARG CACHEBUST=1
When you need to rebuild with selected cache, run it with --build-arg option
$ docker build -t your-image --build-arg CACHEBUST=$(date +%s) .
then only layer below ARG command in Dockerfile will rebuild.

edited Oct 03 '18 at 10:11

krivar

342
1
3
16

answered May 03 '16 at 05:16

VonC

1,262,500
529
4,410
5,250

Thanks for your advise. But it doesn't work for me as I use a cloud CaaS to build the Docker image. There's no way to pass docker arguments. – Raindy May 03 '16 at 08:47
doesn't work for me. Docker version 17.05.0-ce, build 89658be – Stephan Richter Feb 07 '18 at 12:42
@StephanRichter and https://github.com/moby/moby/pull/10682#issuecomment-178794901 ? – VonC Feb 07 '18 at 12:44

score 16 · Answer 3 · answered Sep 07 '18 at 22:05

16

I ran into this same issue myself, and I just decided to use the --no-cache option when I build the image, rather than trying to single out the git repo.

docker build --no-cache -t my_image .

answered Sep 07 '18 at 22:05

tomgsmith99

176
2
6

For my simple use case, this was the simplest/easiest of the proposed solutions. Thank you, @tomgsmith99. – James Adams Jul 29 '19 at 17:50

score 8 · Answer 4 · answered Jan 10 '20 at 16:36

for anyone who has this problem with Gitlab repositories:

Gitlab has this annoying branch id method when calling their API, the ID will appear under your repository name

# this will copy the last change from your brach and it'll invalidate the cache if there was a new change
ADD "https://gitlab.com/api/v4/projects/${PROJECT_ID}/repository/branches/master?private_token=${GIT_TOKEN}" /tmp/devalidateCache

# the actual clone
RUN git clone --depth=1 https://${GIT_USER}:${GIT_TOKEN}@gitlab.com/${git_file_uri} ${BASE_BUILD_PATH}

Michal Zmuda · Answer 5 · 2016-12-28T15:36:54.107

If you use github you can use github API to not cache specific RUN command. You need to have jq installed to parse JSON: apt-get install -y jq

Example:

docker build --build-arg SHA=$(curl -s 'https://api.github.com/repos/Tencent/mars/commits' | jq -r '.[0].sha') -t imageName .

In Dockerfile (ARG command should be right before RUN):

ARG SHA=LATEST
RUN SHA=${SHA} \
    git clone https://github.com/Tencent/mars.git

or if you don't want to install jq

SHA=$(curl -s 'https://api.github.com/repos/Tencent/mars/commits' | grep sha | head -1)

If repository has new commits, git clone will be executed.

score 1 · Answer 6 · answered Aug 31 '20 at 11:11

1

you can also use:

ADD "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h" skipcache

RUN git reset --hard ~Whatever~

as mentioned here: https://stackoverflow.com/a/58801213/8953378

answered Aug 31 '20 at 11:11

Alon Gouldman

3,025
26
29

score 0 · Answer 7 · answered Aug 17 '22 at 06:45

Thanks for the idea of @anq , but the provided api is outdated, the latest api documentation is in Get a reference

use https://api.github.com/repos/{owner}/{repo}/git/ref/heads/{ref}

example https://api.github.com/repos/wind8866/hello-react/git/ref/heads/main

score -3 · Answer 8 · edited Dec 22 '17 at 11:58

-3

For github private repos, you could also pass in your username and password:

RUN git clone -b$BRANCH https://$USER:$PASSWORD@github.com/$USER/$REPO.git $GIT_HOME/

edited Dec 22 '17 at 11:58

Fran Verona

5,438
6
46
85

answered Dec 22 '17 at 04:12

manymoons

25
5

1

I wouldn't like to pass my user password if I'm letting my admins clone just one repo. – Kangur Feb 26 '18 at 13:12

How to prevent Dockerfile caching git clone

8 Answers8

Linked

Related