55

I have a repository on bitbucket that is using LFS. Since using it for some time, I've decided to move the repository back to a space under my control. The only reason I used LFS in the first place was to effectively double my repository size limit (as files in LFS go in a separate bucket) but now I'm moving it, I no longer need to do this.

I need a way to trawl through the entire git history, removing all traces of the work git LFS does (so all files are committed 'normally'). Once this is done, I intend to force push to the new repository.

I've done quite a bit of searching, and come across suggested solutions but I don't understand how to implement/run them because they are high-level.

How do I wave goodbye to git LFS?

Gabriel Staples
  • 36,492
  • 15
  • 194
  • 265
Shadow
  • 8,749
  • 4
  • 47
  • 57
  • That would be something like https://github.com/git-lfs/git-lfs/issues/326#issuecomment-104881826, I suppose? – VonC Feb 09 '18 at 05:33
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/164803/discussion-between-shadow-and-vonc). – Shadow Feb 09 '18 at 05:42

5 Answers5

53

Update current commit only

If you want to move off LFS, but are not so worried about fixing the entire git history, you can do the following;

git lfs uninstall
touch **/*
git commit -a

This will uninstall LFS support, touch every single file (so that git recognises that is has changed) then commit them all. If you like you could be more specific (ie, **/*.png for example). Note that using ** requires extended glob support enabled (shopt -s globstar on bash)

Update entire history

This worked for me - but it throws lots of errors (I think I'm getting an error for every commit that a file hasn't been added to LFS in) and takes a long time (roughly 2-3 seconds per commit).

git lfs uninstall
git filter-branch -f --prune-empty --tree-filter '
  git lfs checkout
  git lfs ls-files | cut -d " " -f 3 | xargs touch
  git rm -f .gitattributes
  git lfs ls-files | cut -d " " -f 3 | git add
' --tag-name-filter cat -- --all

It uninstalls git LFS support (theoretically preventing LFS from messing with the index) then for each commit it makes sure the LFS files are checked out properly, then touches them all (so git realises they have changed), removes the settings for LFS found in .gitattributes so that when cloning it doesn't keep trying to use LFS, then adds the real file to the index.

After you do the above, you will need to do a force push. Naturally, that'll throw anyone else working on your repo into a detached head state - so doing this during a code freeze is wise. Afterwards, it's probably easiest to get everyone to do a fresh clone.

Shadow
  • 8,749
  • 4
  • 47
  • 57
  • I get this error on windows, even while using git for windows: cut' is not recognized as an internal or external command, operable program or batch file. – BikerDude Apr 15 '22 at 16:49
  • 1
    @BikerDude yep, cut is a Linux command. You could use something like cygwin. – Shadow Apr 16 '22 at 21:22
43

git lfs migrate export

From git lfs migrate help:

Export

The export mode migrates Git LFS pointer files present in the Git history out of Git LFS, converting them into their corresponding object files.

Example Workflow

  1. Verify you actually have LFS files with git lfs ls-files.
  2. Remove all filter=lfs lines from ALL the .gitattributes files in your repo. .gitattributes can live anywhere so make sure you find them all otherwise this can cause migration issues later.
  3. Commit any changes you made to .gitattributes.
  4. Make sure you have no changes with git status.
  5. Run the migration: git lfs migrate export --everything --include .
  6. Run git status to make sure you have no changes. If you left .gitattributes with filter=lfs you might incorrectly have changes now.
  7. Verify all the previously listed LFS files are no longer present with git lfs ls-files.
  8. Inspect files (e.g., open formerly LFS files to make sure they aren't corrupt) and run your build to make sure everything works.

Tips

  • Run on case sensitive file system, in case you have file system collisions (e.g. ./LICENSE and ./License) at some point.
  • Git rid of all your filter=lfs lines from ALL your .gitattributes.
  • You may also want to delete LFS remains in .git/hooks directory: pre-commit, post-commit, post-checkout, post-merge.
  • With $GIT_TRACE=1 there should be no sign of ...trace git-lfs: filepathfilter: accepting...
Doug Richardson
  • 10,483
  • 6
  • 51
  • 77
  • 2
    You may also want to delete LFS remains in `.git/hooks` directory: `pre-commit`, `post-commit`, `post-checkout`, `post-merge`. With `$GIT_TRACE=1` there should be no sign of `...trace git-lfs: filepathfilter: accepting...` – pizycki Nov 29 '19 at 14:17
  • @pizycki Thanks. Added your comments to the answer. – Doug Richardson Dec 02 '19 at 00:03
  • 2
    So this was awesome. Worked wonderfully whereas the simple instructions provided in the actual git lfs docs....not so much. Can you explain why the dot after --include? I always thought this had to be some expression but when I used essentially what was in my .gitattributes it didnt work by a long shot. – jkratz Apr 12 '20 at 20:26
  • @jkratz It's a git ``. Note that the `` I used only works if you're at the top level directory in your git checkout. If you want to be able to run the command from anywhere like I intended you can use `:/:` instead. Run `git ls-files ` to see how it works. https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddefpathspecapathspec – Doug Richardson Apr 12 '20 at 20:52
  • Thanks I didn't realize it was a pathspec, thought it was just a glob pattern but rereading the man page it does mention pathspec. thanks! – jkratz Apr 15 '20 at 19:10
  • 2
    What should you do if (e.g. on Github Desktop) it says your local is different from remote by all commits? Should I treat the exported repo as a completely new one? – Tom Charles Zhang Feb 17 '22 at 07:33
  • I got: "batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access." – James Hirschorn Sep 12 '22 at 23:00
  • After doing this and pushing, I got `hint: Updates were rejected because the tip of your current branch is behind hint: its remote counterpart. Integrate the remote changes (e.g. hint: 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details.` when I then tried to pull I got `fatal: refusing to merge unrelated histories`. Not sure what I did wrong. – Pro Q Nov 21 '22 at 03:28
  • When I do this, the `.gitattributes` file is still kept in the initial commit and `git lfs ls-files` still prints a lot of files. – robert May 10 '23 at 09:25
3

For particular file

  1. Remove file/pattern from .gitattributes
  2. Go to file dir and run touch myfile.bin
  3. Commit and push changes
Kamil Kiełczewski
  • 85,173
  • 29
  • 368
  • 345
1

Based on the answer of Shadow I modified entire history update a little. For the ones, who do not want to keep LFS files (which in my case I did not, because they would've been huge).

Use:

GIT_LFS_SKIP_SMUDGE=1 git filter-branch -f --prune-empty --tree-filter '
  git lfs checkout
  git lfs ls-files | cut -d " " -f 3 | xargs rm -f
  git rm -f --ignore-unmatch .gitattributes
' --tag-name-filter cat -- --all

Also it does not fail when no .gitattributes is present in a commit, which again, in my case was not true for all the commits.

Also the repo did not have the original LFS files stored in remote, hence the usage of GIT_LFS_SKIP_SMUDGE=1

This way the repo does not include any LFS references and files, which makes cloning the repo faster and lighter - which was my goal. Large files were used for tests anyway, but since code has improved a lot, running those old tests is irrelevant.

arapEST
  • 491
  • 1
  • 4
  • 16
0

I wrote a script that completely removes lfs from a git repo. WARNING - it rewrites your history.

#!/bin/zsh

git lfs install
git lfs fetch --all
git lfs checkout

lfs_dirs=()

echo "finding lfs files"
for lfs_file in $(git lfs ls-files | sed -r 's/^.{13}//'); do lfs_dirs+=($(dirname "$lfs_file")); done;

echo "found all lfs files"

unique_dirs=()
for unique_dir in ${(u)lfs_dirs}; do unique_dirs+=("$unique_dir/*"); done;

count=${#unique_dirs[@]}

if (($count == 0))
then
echo "No lfs dirs"
exit 0
fi

echo "There are $count unique lfs dirs"

printf -v migrate_paths ',%s' "${unique_dirs[@]}"
migrate_paths=${migrate_paths:1}

echo "performing: git lfs migrate export --include='$migrate_paths' --yes --verbose --everything"
git lfs migrate export --include="$migrate_paths" --yes --verbose --everything
git lfs uninstall
MidasLefko
  • 4,499
  • 1
  • 31
  • 44
  • Thanks for the response. As per the stackoverflow guidelines, link only answers such as this are discouraged, as links can break. Would you consider copying your script into your answer? – Shadow Mar 20 '23 at 12:41