66

How to delete a never-use-again file and release the storage back to the github lfs quota?

Does deleting the reference point of the file in git history work in this situation?

CodeWizard
  • 128,036
  • 21
  • 144
  • 167
lucemia
  • 6,349
  • 5
  • 42
  • 75
  • Related: see [my notes here about why I now think using `git lfs` is bad](https://stackoverflow.com/q/75946411/4561887). This Q&A, revealing the extreme lack of support on GitHub to stop using `git lfs`, confirms that. – Gabriel Staples Apr 07 '23 at 17:58

5 Answers5

34

There doesn't currently appear to be a great way of removing large assets from git-lfs. GitHub's current suggestion is to use a tool called The BFG to completely strip all existence of the file from your repo.

Presumably it'll then be removed from the lfs storage when git's garbage collection is next run by GitHub.

For more info see https://help.github.com/en/github/managing-large-files/removing-files-from-git-large-file-storage

Edit 2019-11-20

GitHub has updated their documentation on this explicitly stating there is no way to free up your git-lfs storage without deleting your repository. https://help.github.com/en/github/managing-large-files/removing-files-from-git-large-file-storage

Their only recommendation is to still use the BFG tool to strip the file from your repository, which will reduce the size of your repository when cloning, but will still count towards your git-lfs quota until you delete the repository.

GitLab again suggests using the BFG tool and will automatically clean the lfs storage of any files which aren't referenced in any commits, although currently this doesn't seem to be working correctly. This issue has been open since 2017 with no resolution.

BitBucket has a section in the repository settings which you can clean up lfs files. https://www.atlassian.com/git/tutorials/git-lfs#deleting-remote-files

Gabriel Staples
  • 36,492
  • 15
  • 194
  • 265
Wader
  • 9,427
  • 1
  • 34
  • 38
  • 14
    It should be noted that all of your history will change which means all of the commit ids will change everywhere. This _can_ have drastic issues if your repo pulled by others and they are trying to contribute back. – g19fanatic Jun 23 '16 at 19:31
  • 16
    Not being able to purge old files from Github's LFS storage is a serious gotcha! It doesn't take long for a 500mb file to eat 50 gigs of paid quota, since each change to the file is kept forever. The only "solution" is to obliterate the entire repository! – mpoisot May 14 '20 at 17:08
  • 1
    After completing the steps on Github afterward to delete/re-create the repository, any of the LFS files that I chose to keep in the repo would NOT upload again to the remote Git LFS storage. Subsequent `git clone`s would fail with a 404 upon attempting to download the LFS object. The objects appeared fine on the source local repo using `git lfs ls-files`. To fix, I had to `git rm` and commit the LFS files I wished to keep, then re-add `git add` them... Only then would `git push` send the files to remote LFS storage. Make sure to cache the files first so you do not destructively remove them. – pztrick May 25 '20 at 05:55
  • This "solution" on GitHub is crazy. – Gabriel Staples Apr 07 '23 at 17:47
19

The answer given by Wader is not related to the question.

The given link to a github article is about large files directly checked into the git repo, which is NOT the same as git-lfs!

As for the question: how this situation is handled depends on the underlying storage server. git-lfs support can be provided by different services, e.g. GitHub oder GitLab.

GitLab stores git-lfs files not per repository. They are stored by their SHA-256 hash value. If the same file is used in several repositories (e.g clones), it is only stored once. It solely depends on GitLab how it handles referenced files and should be looked up in the manual.

According to GitHubs documentation (https://help.github.com/articles/removing-files-from-git-large-file-storage/), it is recommended to delete and recreate a repository: "To remove Git LFS objects from a repository, delete and recreate the repository. When you delete a repository, any associated issues, stars, and forks are also deleted."

Chris
  • 316
  • 2
  • 5
9

The answer given by Chris is not accurate; combined with the comment from 0xcaff ("that's a terrible solution"), it may mislead the inexperient user (or distracted reader).

The right answer is a mixture of Wader's and Chris'.

I just solved a case where I wanted to mirror a repository (let's call it REPO-1) that once upon a time used LFS, but has not been history-cleaned from the large files.

At some moment in the last month I removed support for LFS in REPO-1 with git lfs uninstall. And that was working fine for subsequent git clones (i.e., no git-lfs anymore). But today when I tried to pull a previously push --mirror version of REPO-1 I got messages Smudge error: Error downloading <big-filename> ... Object does not exist on the server: [404] Object does not exist on the server

So, if you want to get rid of LFS and any reference to the once-tracked files you have to:

  1. uninstall LFS from the repo;
  2. clean the repo's history;
  3. delete (or rename) the old (github) repo, create a new one and push the cleaned repo into it.

1

To uninstall LFS is a simple command like git lfs uninstall will do it.

2

To clean the repo's history is a bit scary, but works just fine with git filter-branch, item 3 of the Github docs --https://help.github.com/en/articles/removing-sensitive-data-from-a-repository -- should be about enough:

$ git filter-branch --force --index-filter \
      "git rm --cached --ignore-unmatch FILENAME-1 FILENAME-2 FILENAME-3" \
      --prune-empty --tag-name-filter cat -- --all

3

The deletion and re-creation of the repository is not as destructive at seems like, remember that your local repository is completely independent, self-sustained clone of what's in Github. When you push the your local (clean) version to the new, blank repository it will be like that (deletion/creation) never happened.

Brandt
  • 5,058
  • 3
  • 28
  • 46
  • 19
    Sure the code stays intact, but the fact remains that deleting the repo on Github means you lose all the stars, issues, forks, etc. That's pretty harsh! The free and $5/month tiers are pretty small (1gb and 50gb), so if you actually need LFS then you will hit that wall, then discover the only option is nuclear. – mpoisot May 14 '20 at 17:17
  • 3
    True story, @mpoisot, thanks for pointing that out! – Brandt Jul 13 '20 at 15:13
  • will this cause the deletion of all git history? or just the relevant commits (that include the large files) from the history? – Immanuel Aug 29 '21 at 08:33
  • That should not delete the history, no. Will rewrite the history, sure. – Brandt Aug 30 '21 at 09:43
3

I know deleting the repo isn't the best option and I can't believe that this is the solution recommended by GitHub in 2022. As pointed out in GitHub documentation:

To remove Git LFS objects from a repository, delete and recreate the repository. When you delete a repository, any associated issues, stars, and forks are also deleted. For more information, see "Deleting a repository." If you need to purge a removed object and you are unable to delete the repository, please contact support for help.

If your problem is only the issues created in your original repository you could migrate them to the new recreated repo before deleting the original, as pointed in this post.

The GitHub UI lets you migrate single issues like so:

gh issue transfer <issue ID> <destination repo>

But that would take hours if some of the repos had over a hundred issues (also my case). Combining it with a few shell pipes let us to:

gh issue list -s all -L 500 --json number | \
    jq -r '.[] | .number' | \
    xargs -I% gh issue transfer % https://github.com/<destination repo>

All this will do is grab the last 500 issues from the original repo and transfer them to yours new <destination repo>. If you have to transfer more than 500 issues increase this value or just run it multiple times. Paid attention because the command must be executed in the correct original repo directory.

So far so good, after that you could purge the original repo and remove LFS storage from GitHub.

Be careful and remember that when you delete a repository stars and forks are also deleted, it wasn't my case but could be yours. If so, search for complementary solutions before blindly removing your repo.

Gabriel Braico Dornas
  • 439
  • 3
  • 11
  • 6
    Unbelievable. A single bad push of a too-large LFS file and now my repository is permanently locked unless I either pay them $5 a month in perpetuity or I delete and migrate the entire repository. What an incredibly consumer-unfriendly and exploitative business practice. – puzzl Jan 27 '23 at 17:23
2

Founds this new method (2023)

enter image description here enter image description here

mgear
  • 1,333
  • 2
  • 22
  • 39