0

I'm working in a monsterously huge git mono-repo on the order of 100 GB in size. we have a git post-checkout hook in .git/hooks/post-checkout which contains the following hook to run git lfs after each checkout:

#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/post-checkout.\n"; exit 2; }
git lfs post-checkout "$@"

I just ran git checkout main and after literally 3 hrs of running and 27 GB of data downloaded through my internet connection, it failed because my disk was full, at about 97% complete through the git lfs post-checkout hook operation.

So, I cleared up some disk space.

Now, git checkout main fails with error:

error: Your local changes to the following files would be overwritten by checkout:

So, I tried running git lfs post-checkout main (I don't even know if this is a reasonable command--I'm guessing here) manually, and it fails too, with:

This should be run through Git's post-commit hook. Run git lfs update to install it.

Is there any way to resume my git lfs operation so I do NOT have to clear all 27 GB of data just downloaded and start downloading it all over again from scratch (via git reset --hard && git clean -fd && git checkout main)?

Note that git checkout main had shown some errors like this as a result of the git lfs post-checkout hook operation:

error: Your local changes to the following files would be overwritten by checkout:
    [list of tons of files]
error: The following untracked working tree files would be overwritten by checkout:
    [list of tons of files]
Aborting
0
Gabriel Staples
  • 36,492
  • 15
  • 194
  • 265
  • Try `git reset --hard` and then `git checkout main` again. – bk2204 Jul 28 '21 at 00:23
  • @bk2204 that works, but `git reset --hard` deletes all 27 GB downloaded and makes `git lfs` start from scratch, downloading the entire thing again. – Gabriel Staples Jul 28 '21 at 00:58
  • `git status` currently shows tens of thousands of files as "Changes not staged for commit". – Gabriel Staples Jul 28 '21 at 01:00
  • 1
    Yes, Git-LFS is annoying like that. I don't actually use it and don't know what the right workaround is: you need to get it to start up the long-running filter and do the filtering of all the files that `git checkout` was in the middle of checking out. (The details will depend on your Git and Git-LFS vintage.) It's probably literally faster and easier to just re-run the entire 30 GB checkout. – torek Jul 28 '21 at 02:41

1 Answers1

0

One idea I had is to comment out the last line in the .git/hooks/post-checkout dir so that it looks like this:

# git lfs post-checkout "$@"

Then, save the file and run git checkout main. When done, run git lfs pull, then uncomment the line above to return that file to normal. This way, since the git checkout alone takes like 10 minutes and checks out like 80000 files, at a minimum, that part will get done and pass before the git lfs part runs and takes forever (3+ hrs) and possibly crashes.

I'm not sure if that works or not, but anyway, I don't have an exact solution to my problem, so here's what I did:

  1. git checkout main failed due to my disk running out of space. git status now shows thousands of files which are changed and unstaged. So, run the following to clear up your git status in preparation to run git checkout again:
    git status        # see that there are thousands of unstaged changes
    git reset --hard  # reset changed files back to how they are in `main`
    git clean -fd     # delete all unstaged files and directories
    git status        # this should now be clean
    
  2. Reduce usage on your disk by deleting your bazel cache, if using the Bazel build system:
    df -h  # check current disk usage
    rm -rf ~/.cache/bazel  # delete Bazel build cache
    df -h  # ensure you have more space free now
    
  3. Reduce the size of your .git folder to free up disk space. See my full answer here on this: How to shrink the .git folder.
    time git lfs prune
    time git gc
    time git prune
    time git repack -a -d --depth=250 --window=250
    
  4. Now try the checkout again, just letting it re-download the entire 30 GB or so over the course of 3+ hrs. :( Oh well...at least it will pass now since I just freed up disk space by 1) reducing the size of my .git dir and 2) deleting my Bazel build cache:
    time git checkout main    
    

Done.

Gabriel Staples
  • 36,492
  • 15
  • 194
  • 265