0

I have an enormously large folder. In this folder are a lot of videos and small files scattered around. I value the small files, not the videos.

I want to back up all the small files onto Github. If I directly try to upload the folder onto Github, I will get multiple large fille (>100 MB files) errors. The videos and small files are scattered around everywhere, making it difficult to remove videos one by one.

I know how to remove a single giant file from Github: Git error, need to remove large file

However, is there an easy way to do this for all the large files in my folder, such that I can push everything else onto Github with ease?

Note: I don't want to use cloud storage, because all I want to store are my small files.

Daniel A. White
  • 187,200
  • 47
  • 362
  • 445
Galen BlueTalon
  • 173
  • 4
  • 20
  • 2
    *I want to back up all the small files onto Github.* GitHub is **not** a backup solution. – Daniel Mann Jan 14 '22 at 20:41
  • To be sure we understand, you want to have only your small files in the repository? Large files will only live on your disk? – Schwern Jan 14 '22 at 20:44

1 Answers1

1

If you want to retain these large files locally, you first delete them from Git, but retain them in the working copy.

  1. Find them all with find . -size +100M -not -path './.git/*'
  2. Remove the files from Git, but keep the file itself: git rm --cached.
  3. Add them to .gitignore.

You can combine steps 1 and 2: find -X . -size +100M -not -path './.git/*' | xargs git rm --cached.

I would recommend using this as an opportunity to move all your large assets to a single directory and ignore that directory.

Now that they're deleted from Git and safely ignored, use the BFG Repo-Cleaner to delete them from history.

bfg --strip-blobs-bigger-than 100M --no-blob-protection my-repo.git
Schwern
  • 153,029
  • 25
  • 195
  • 336
  • To clarify for #3 - how do I add them to .gitignore? – Galen BlueTalon Jan 16 '22 at 02:34
  • @GalenBlueTalon You can [append the output of the find command to .gitignore](https://stackoverflow.com/questions/6207573/how-to-append-output-to-the-end-of-a-text-file). If possible, first move those files to directories for large assets (like assets/videos/) and ignore just the directories. – Schwern Jan 16 '22 at 19:35