4

I have a git repo where many files should be lfs because they are larger than 100M.

Looking around I was unable to find a step by step guide that explain how to migrate a real existing repo with many branches and where lfs files are within subdirectories.

In my case large files are spread around the repo like this:

code/track1/file000.pkl
code/track3/dat000.bin
code/track4/pip000.pkl
code/subcode/track5/pip000.pkl
code/subcode/track5/pop000.model

I suppose to convert the git project into git lfs just using git lfs migrate:

git lfs install
git lfs migrate import --include="*.pkl"
git lfs migrate import --include="*.bin"
git lfs migrate import --include="*.model"
git commit -m "migrating models"

but this does not do anything.

nothing to commit, working tree clean

I want convert all the repo, I mean all the files, the history and also all existing branches.

In other words, git lfs migrate seems to be stable now but not so user frendly.

Running git lfs track "*.pkl" seems to have an effect:

modified:  code/track1/file000.pkl
modified:  code/track4/pip000.pkl
modified:  code/subcode/track5/pip000.pkl

but what to do next. I see that git lfs track will begin tracking a new file or an existing file that is already checked in to your repository.

But what about the history? I'm struggling because I don't want end up with a messed repository that I have to reimport from scratch or where have to deal with filter-branch..

freedev
  • 25,946
  • 8
  • 108
  • 125
  • A quick search points to [this tutorial](https://docs.gitlab.com/ee/topics/git/lfs/migrate_to_git_lfs.html). As that roughly what you're searching for? – Joachim Sauer May 15 '20 at 10:15
  • I also read that git-lfs-migrate should be preferred over bfg. Because BFG writes gitattributes files incorrectly if you're updating multiple file types: [rtyley/bfg-repo-cleaner#116](https://github.com/rtyley/bfg-repo-cleaner/issues/116) – freedev May 15 '20 at 10:21
  • well, if you don't want to use filter-branch and don't want to use bfg, then you're running out of options ... – Joachim Sauer May 15 '20 at 10:23
  • You're implicitly saying that there is no way to do it with git lfs migrate. – freedev May 15 '20 at 10:24
  • 1
    I think I misread the lfs migrate docs. **By default** it only migrates local-only changes and nothing that's already on the remote. You need the correct `--include-*` switches to do "history-rewriting" migrates. `--everything` might be what you need. – Joachim Sauer May 15 '20 at 10:36
  • Thanks, this is interesting. I wonder why there was not even mentioned in many posts. But what to do next, `push --force --full` ? – freedev May 15 '20 at 10:43
  • At the moment, both BFG and git lfs migrate have limitations / bugs according to end-users' expectation. Blobs are "missed", probably because of "unexpected" references. – Yves Martin May 18 '20 at 08:41

1 Answers1

6

Git LFS provide a tutorial on how to use git lfs migrate

A few things to mention:

  1. When you are using lfs migrate, it goes through and changes the history, such that when you call git lfs migrate import --include="*.pkl", it adds that to the .gitattributes to be tracked under LFS, and migrates .pkl files to LFS throughout your history. As such, you don't need to call git commit afterwards. Note, this would be different if you call git lfs import migrate --no-rewrite, which as the docs indicate leads to a separate commit in which the files are migrated to LFS. (But, given what you've indicated in your question, I don't think that's what you want.)
  2. You indicate that you are working in repo with multiple branches. As such, you might want to use --everything, which the docs indicate includes all local and remote branches in the migration.
  3. If you're unsure as to whether the LFS migration has worked, you can check the lfs directory to see if objects have been moved to LFS storage, or run git lfs ls-files to see which files are currently tracked and stored under LFS. Also, you might want to run the LFS migration with --verbose, which will print out the specific files that have been moved to LFS storage as it progressed through the migration.
amaidment
  • 6,942
  • 5
  • 52
  • 88
  • For me `migrate` didn't work. I had to remove files from tracking, then install `LFS` and add them again. – kelin Sep 01 '22 at 20:12
  • Regarding step 2: This doesn't work. It only includes local branches. You have to mirror the remote repository. – wlfbck Mar 22 '23 at 11:08
  • step 2 should work on all branches, but requires a `git push --force --all` to push the changes back to all affected remote branches (see: https://github.com/git-lfs/git-lfs/issues/4264) – Seth Jul 12 '23 at 00:11