Remove all information from a .git directory that can be re-downloaded

Question

I have a git repository that, when just checked out, takes around 2.3 GiB even in the shallowest configuration, of which 1.9 GiB is inside .git/objects/pack. The working tree files are just about .5 GiB.

Considering I have a remote from which I can re-fetch all the objects if needed, the question is:

• What (and how) can I delete from inside .git everything that I could then re-fetch safely, with simple git commands, from the remote?

Testing a bit, I found out that if I delete everything under .git/objects/pack/, it will be re-downloaded from the remote with a simple git fetch.

There are some complaints like:

error: refs/heads/master does not point to a valid object!
error: refs/remotes/origin/master does not point to a valid object!
error: refs/remotes/origin/HEAD does not point to a valid object!

But then .git/objects/pack gets repopulated and further calls to git fetch don't complain anymore.

Is it safe to nuke .git/objects/pack* like this?

Assumptions:

• There are no local-only commits in the repo or any form of git manipulation (like adding/removing objects from the stage), just checking out a specific branch in shallow mode.
• The remote won't be rewriting history for the checked out branches.
• I have no control whatsoever over the contents of the remote repository itself. It's a dependency of my project, but a fast changing one that is only available as git, and I want instructions for automated use in a continuous integration setting. Tips on how to modify the repository itself to make it take less space aren't going to help.
• As I mentioned earlier, 1.9 GiB is for a shallow clone of the one branch I'm interested. It's a lot bigger than that when it's non-shallow, due to it's long history (open-source project that has over 10 years).
• There are other repositories checked out in the same continuous-integration pipeline and I'd like to apply the same reduction of redundant-with-remote info in all of them.

The intent is to reduce as much as possible the amount of space taken by artifacts from a continuous-integration pipeline, but retaining enough information so that a those artifacts could be downloaded and restored to working order in the developer workstation with as little (and as normal) commands as possible.

Answer 1

Deleting stuff in .git/ is just going to break things. That contains the complete history of your project and those pack files are how git saves space. There are far, far better ways to reduce the size of your repo.

First is to run garbage collection, git gc. This will do a number of things to reduce the size of the repository on your disk. You shouldn't have to, this runs periodically, but it might help.

If it doesn't, try a shallow clone where you only get part of the history. This only clones the latest 100 commits from master.

git clone --depth=100 <remote>

Similarly, you can just clone one branch.

git clone --single-branch --branch master <remote>

These can be "deepened" later with git-fetch.

But the best thing to do is to reduce the size of your repo. Git is very efficient on space and 2 gigs is enormous. It suggests there are a lot of very large binary files in the repository, images, videos, spreadsheets, and compressed files... which git cannot efficiently compress. To handle this there are two tools, git-lfs (Large File Support) and the BFG Repo Cleaner.

git-lfs lets you store old versions of large files in cloud storage rather than in everyone's .git directory. This can immensely reduce the size of the repository... going forward.

BFG Repo Cleaner lets you easily rewrite history, including options to remove large files.

Put them together, and you can use the BFG Repo Cleaner to change existing large files to use git-lfs. This can immensely reduce the size of your repository. For example, this would change all *.mp4 to use git-lfs.

$ java -jar ~/bfg-1.12.15.jar --convert-to-git-lfs '*.mp4' --no-blob-protection

Instructions for that can be found here.

The other important thing is to not compress files. You mentioned continuous integration artifacts, and I'm willing to bet they're compressed. Git will do its own more efficient compression, and it can ensure there's only ever one copy of a file through history, but it can only do it on text. Unpack tarballs and zipfiles before committing them.

---

If you absolutely cannot reduce the size of the repository, your remaining option is to have everyone share one .git directory. You can do this with the --git-dir option, or by setting GIT_DIR.

git --git-dir=/path/to/the/.git log

This is a terrible idea. While everyone can have their own checkout, they'll all be sharing the same repository state. If one dev makes a change, the other devs will see it, but now with a different working directory.

For example, dev1 adds a file.

$ touch this
$ GIT_DIR=~/tmp/foo/.git git add this
$ GIT_DIR=~/tmp/foo/.git git st
On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

    new file:   this

Then dev2 suddenly sees this.

$ GIT_DIR=~/tmp/foo/.git git status
On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

    new file:   this

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    deleted:    this

They're sharing the same staging area, but not the same working copy. Devs will be stumbling over each other constantly.

---

If git clone --depth=1 is still producing repos that are too big, then there's simply a lot of data in each checkout. There isn't much you can do about that. If .git on a shallow clone is 2 gigs, then the checkout is going to be even larger.

As for the idea of performing surgery on .git, maybe you could get away with deleting some objects and hope that a git fetch --deepen can fix it, but maintaining this across multiple devs... it's a maintenance nightmare.

At that point, you might as well just delete .git entirely. Now you've effectively exported the latest commit. There's various ways to do this directly.

Or just stop wasting time and money and buy bigger hard drives. Every person-hour spent on this is a hard drive you could have bought.

Answer 2

reduce as much as possible the amount of space taken by artifacts from a continuous-integration pipeline, but retaining enough information so that a those artifacts could be downloaded and restored to working order in the developer workstation with as little (and as normal) commands as possible

I don't fully understand your case but one often forgotten way of reducing network data size and server's memory use is to:

• distribute some stable repository (which includes only branches which are not rewritten), and then
• use use --reference <path> while cloning.

In normal development conditions (text files, not all fof them updated in each commit) it's way more efficient than using shallow clones.

As for the thing you asked, I think it makes no sense to try to save on removing anything from the repository. Most data is used for pack, which is needed and rest is insignificant.

PS: the repository can be initialized in temporary storage just by git itself:

CACHE_REPO=/tmp/repo
if ![ -d "$CACHE_REPO" ]; then
  git clone --single-branch --no-checkout --branch=_BRANCH_ _REMOTE_ "$CACHE_REPO"
fi

_BRANCH_ is master or some other branch you are sure will not be force-pushed. You can try making it shallow, it might or might not work, I'm not sure about it.

Answer 3

• What (and how) can I delete from inside .git everything that I could then re-fetch safely, with simple git commands, from the remote?

How about everything?

If you don't want to worry about the internals of .git and whether something is or not recoverable, you can save just enough information to check it all out again, and restore the workspace to a functionally similar state than it was when running in the C.I. pipeline.

In the C.I. Pipeline

Add somewhere a file like this (let's call it degit.sh)

#!/bin/bash
set -ex
GIT_REMOTE=$( git remote get-url origin )
GIT_BRANCH=$( git rev-parse --abbrev-ref HEAD )
GIT_COMMIT=$( git rev-parse HEAD )

# TABs, not spaces, indenting the block below:
cat <<-EOF > .gitrestore
    set -ex
    test ! -e .git
    tmpclone=\$( mktemp -d --tmpdir=. )
    git clone $GIT_REMOTE -n --branch=$GIT_BRANCH \$tmpclone
    ( cd \$tmpclone ; git reset --hard $GIT_COMMIT )
    mv \$tmpclone/.git .
    rm -rf "\$tmpclone"
    rm -f \$0
EOF

rm -rf .git

Then, inside the root of each of the git repos of your Continous Integration (C.I.) workspace, you call it so that it generates a .gitrestore file.

It will look something like this:

set -ex
test ! -e .git
tmpclone=$( mktemp -d --tmpdir=. )
git clone git@example.com:example/repo.git -n --branch=example-branch $tmpclone
mv $tmpclone/.git .
git reset --hard example-commit-hash
rm -rf "$tmpclone"
rm -f $0

Notice that it self destructs after running successfully. You don't want to run it twice.

In the Developer Machine

Now your developer can fetch the C.I. artifacts and run, inside each repository:

bash .gitrestore

And it will have a repository that looks very much like what the C.I. pipeline had, except for an updated view of the remotes, which allows the developer to compare what the C.I. had with what she has.

Other considerations

This assumes that only the C.I. machine is constrained for space, not the developer machine (neither her bandwidth).

If you want to save space/bandwidth on the developer end, you can pass --depth=1, which will clone only the specified branch (i.e., it implies --single-branch and will restrict the history to a single commit.