122

I have a project where I stored video files with Git LFS. Now I ran into some complications with my build server that doesn't yet support Git LFS. As it's an external service, I can't really affect the build process, and thus would want to move the files from under Git LFS back to "regular" Git. I managed to untrack the file types with git lfs untrack '<file-type>' but git lfs ls-files still gives a list of the files previously added.

I imagine I could remove the files, push the changes and then manually re-add them, but is this really the recommended way of doing things?

Olli Niskanen
  • 1,535
  • 2
  • 12
  • 18

6 Answers6

169

I have just recently run into this problem where assets were accidentally added to git-lfs on one branch that shouldn't have been. My solution was:

git lfs untrack '<file-type>'
git rm --cached '<file-type>'
git add '<file-type>'
git commit -m "restore '<file-type>' to git from lfs"

The result is a rewrite of the git-lfs oid sha256 pointers with the standard file contents.

(Edit 2019-03): The accepted answer was changed to provide an easy solution for simpler cases. See also the edits in the answer by VonC for alternate solutions in case you have a more complex case on hand.

Olli Niskanen
  • 1,535
  • 2
  • 12
  • 18
mred
  • 1,814
  • 1
  • 11
  • 5
  • 7
    With `.gitattributes` is set up correctly to track the files I want, this solution worked beautifully. Much simpler than some of the other solutions posted. – Josh Rickert May 17 '17 at 21:15
  • 1
    This worked wonderfully. Should be listed as the true answer. – Aditya May 20 '18 at 12:11
  • 3
    worked for me in '18. tip for others: 'file-type' was '**.jpg' – oligofren Dec 28 '18 at 17:19
  • 1
    Based on the upvotes, I'll change this to the accepted answer. – Olli Niskanen Mar 13 '19 at 11:40
  • 3
    @OlliNiskanen while it is gratifying to be marked as the accepted answer, I feel it important to note that while this is a good and simple solution to the specific question asked, for more complex cases readers should consult the batching methods suggested in the edits by tstephens619 and ttaylorr to VonC's answer. – mred Mar 14 '19 at 13:22
  • 2
    @mred, Agreed. I suggested an edit that links to the previous accepted answer. This answer was pretty buried and people found it helpful, so I feel this should stay as the first thing visitors see. – Olli Niskanen Mar 14 '19 at 17:34
  • This doesn't work for me. I tried 'git lfs untrack ', then 'git rm --cached ' etc. The 'git commit' command doesn't find anything to commit. I also tried 'git add --force', didn't make a difference. I have git 2.25.1. – fencekicker Sep 06 '22 at 09:13
  • 1
    @fencekicker I think a missing step from this is that you have to already have the normal file (not the lfs pointers) in your directory. Try doing `git lfs checkout ` and then following the steps here. – hackerb9 Dec 02 '22 at 19:45
  • 1
    @hackerb9 it was something else, as far as I remember; we had a wildcard rule and a more specific rule for LFS files. At first I had removed only the specific file rule, but there was still a wildcard rule (thing /* vs /*.bin). I had to remove the '*' rule and add specific rules for the files to keep under LFS, and then the conversion worked for the other files. – fencekicker Dec 05 '22 at 14:10
74

As of Git 2.16 (released Jan 17th, 2018), you can do this easily with the --renormalize flag of git add:

git lfs untrack "<pattern>"
git add --renormalize .
git commit -m "Restore file contents that were previously in LFS"

From Git's documentation:

--renormalize: Apply the "clean" process freshly to all tracked files to forcibly add them again to the index. This is useful after changing core.autocrlf configuration or the text attribute in order to correct files added with wrong CRLF/LF line endings. This option implies -u.

The key part here is "all tracked files". Normally, filters are only run when a Git operation changes a file in the work tree. Changing the LFS whitelist in .gitattributes isn't a Git operation, and so the index ends up in an inconsistent state after you run git lfs untrack. Running git add --renormalize . tells Git to re-run filters on every file in the repository, which ensures that all files which should be in LFS are—and that all files which shouldn't be aren't.

Tom Hebb
  • 991
  • 1
  • 8
  • 14
  • 4
    The good thing about this method is that if you checkout from the past or another branch, the old lfs stuff is still there. But going forward it is not using lfs any more. The one thing I would add, is to remove the lfs stuff now from .gitattributes (or in my case deleting the whole file) and checking that in too. – David Casper Jun 27 '19 at 01:25
  • 3
    It should be noted that this won't change the history, meaning that if you check out an older commit, you might get pointers to LFS instead of the actual files. And if you have since moved the repo or deleted the LFS, you won't get those old large files back this way. – Thomas Tempelmann May 11 '20 at 14:03
  • The bad thing about this method is that it touches ALL files and you will end up with a massive commit of everything. – BigMiner Nov 02 '21 at 16:42
  • 1
    The double quotes should be used for the Windows command line – Serge Sotnyk Jan 20 '22 at 11:59
34

Issue 641 mentions the same issue.

I tried to stop using Git LFS, but found no way to revert my previous tracked pointer files using git lfs uninit, git lfs untrack, git rm... after I move those files back it still lists as tracked by Git LFS with git lfs ls-files, how can I opt out the whole Git LFS stuff from my repo?

The answer was:

  1. Remove all filter.lfs.* git config entries with git lfs uninit.
  2. Clear any any attributes that use the lfs filter in .gitattributes by running git lfs untrack for each file type, or deleting .gitattributes if LFS is all you ever used it for.

After this, any added files will go straight to git.

But this was not so simple:

I later end up LFS pointer files in my working directory and have to recover all my pictures from .git/lfs using the sha1 hash stored in those pointers manually.

Update March 2016, the issue 957 illustrates a possible solution by tstephens619:

I made the same mistake of including several small graphics formats into my git lfs tracking list.
I was able to move this files back into git by doing the following:

  • Create a list of all of the files currently being tracked by git-lfs, filter out *.gz and *.rpm (I want to still track those extensions with git-lfs)

    git lfs ls-files | grep -vE "\.gz|\.rpm$" | cut -d ' ' -f 3 > ~/temp/lfs-files.txt

  • Stop tracking the small graphics files

    git lfs untrack "*.tts"
git lfs untrack "*.bfx"
git lfs untrack "*.ttf"
git lfs untrack "*.xcf"
git lfs untrack "*.pkm"
git lfs untrack "*.png"

  • Temporarily uninit git-lfs

    git lfs uninit
# Git LFS 2.x+
git lfs uninstall

  • Use the file list to touch each file:

    cat ~/temp/lfs-files.txt | xargs touch

git status will now show each file as modified

  • Add the changes to git index (I did this via git gui)

  • commit the changes and then re-init git-lfs

    git commit
git lfs init

The maintainer ttaylorr adds:

One way to do this would be:

for file in $FILES_TO_REVERT; do
  git lfs untrack "$file";
  git rm --cached "$file";
  git add --force "$file";
done

git commit -m "..."

My preference would be not to add a command to Git LFS to the above effect, since it is possible in a number of different way with the porcelain commands provided by Git and Git LFS

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • 4
    Thanks for finding the correct issue from the project. Still, even this solution requires moving the files manually. Seems sort of odd when the workflow from Git to Git LFS is basically `git rm --cached ` -> `git add ` -> `git commit` as long as you have the tracking set up correctly. – Olli Niskanen Jan 28 '16 at 16:11
  • 1
    @Klipi I agree: it seems the untrack scenario is still perfectible. – VonC Jan 28 '16 at 16:17
  • 1
    @Klipi Good call. I will monitor it. – VonC Jan 28 '16 at 16:54
  • 5
    My version of LFS (2.0.2) doesn't have a `uninit` command. Instead of `git lfs uninit` I had to use `git lfs uninstall`. – endavid Apr 29 '17 at 09:23
7

EDIT: After a couple of years of using GIT LFS successfully, and a number of up/down votes on this answer, I think this warning still applies: GIT LFS has a lot of flaws, such as difficult management of which files should be in LFS, performance issues when (accidentally) adding many small files to LFS on Windows, limited support for multiple remotes and remote URL formats, difficulty removing files from LFS, various problems you can run into when merging, etc. GIT LFS is a foreign element in GIT that exists outside the revision tree. I would, however, like to rephrase my original warning as follows:

  1. Only put files into GIT LFS that you would normally put into GIT (e.g., "source files" that you own and change occasionally)
  2. Only put large files into GIT LFS.
  3. If you need a system for managing binary dependencies, consider using a package manager instead.
  4. Don't use Subversion as a replacement for GIT LFS. It is worse.
  5. Be prepared to mess up your working directory. Make sure to back up (push) valuable changes before you do any major changes to LFS.
  6. When merging, always merge .gitattributes first.

EDIT: Here is my original answer:

It is difficult remove anything from GIT LFS, and although the solutions presented here may work (with modifications), they require a lot of effort and may have side effects on your repository.

If you arrived here, it is time to ask yourself whether you want to manage your large files with GIT LFS and whether GIT itself (which is inherently bad at managing large files, because it is a distributed version control system) was a good choice.

If you have many large files and you are a single organization working on your project, something like Subversion may work better for you.

Florian Winter
  • 4,750
  • 1
  • 44
  • 69
  • 7
    @ericfrazer Every single visit to this question proves my point, as following the official documentation of GIT LFS, removing files from LFS should be a no-brainer. Except, it isn't, because it does not work. People are struggling, because of a combination of a) it is hard to fine-tune what files to add and b) adding many small files has performance issues on Windows because GIT LFS runs as a sub process, and starting sub processes is expensive without fork(). Bottom line: For your own mental sanity, don't use LFS before all of this is fixed. Which hasn't happened for years now... – Florian Winter Dec 18 '18 at 11:53
  • This answer is more a point of vue than an answer, But as question stand for using GIT on large binary files, This look more like a **Warning**, I vote to **not delete** this answer. – F. Hauri - Give Up GitHub Sep 20 '21 at 08:31
5

I had problems doing steps in Windows. To remove all git lfs tracked files and restore original file I did the following in git bash:

  1. Removed .gitattributes

  2. git lfs ls-files | cut -d ' ' -f 3 > lfs-files.txt

  3. Execute following snippet:

Snippet:

while read file; do
  git lfs untrack "$file";
  git rm --cached "$file";
  git add --force "$file";
done <lfs-files.txt
Joker
  • 2,304
  • 25
  • 36
  • What does the `-f 3` sequence? – Francisco Maria Calisto Sep 30 '21 at 13:09
  • 1
    @Francisco Maria Calisto It splits the result of git lfs ls-files on spaces and takes the 3. entry (which is the file name). See https://unix.stackexchange.com/questions/122055/what-constitutes-a-field-for-the-cut-command for a wider explanation of `cut -f` – Joker Oct 01 '21 at 22:11
1

I tried and was able to use the following command successfully to revert my repo back to a regular one according to their official doc:

git lfs migrate export --include="*.psd" --everything

The link to the original official doc is at https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-migrate.adoc#migrate-local-history.

Hope this is of use to anyone in the same situation!

Kris Stern
  • 1,192
  • 1
  • 15
  • 23