15

I have a project versioned with Git that I'd like to make open source, but it has some private information in it that is specific to the environment in which it was originally used. I'm going to change the information in question to load from a config file which is not included in the repository. I realize I should have done this in the first place, but since the private information still exists in previous commits, how can I go about removing it from my history? Do I just have to start a new repository based on the latest commit and lose all my history or is there a way to salvage the current repository while removing any record of the private information?

Edit: To clarify, I don't want to completely remove the files that contain this private information, because they are still used. Rather, I want to remove/blank out/change the occurrence of certain strings within them.

Roberto Tyley
  • 24,513
  • 11
  • 72
  • 101
Jimmy
  • 35,686
  • 13
  • 80
  • 98

2 Answers2

6

I'd recommend using the BFG Repo-Cleaner, a simpler, faster alternative to git-filter-branch specifically designed for removing private data from Git repos.

The usage instructions give the steps in more detail, but the core bit is just: download the BFG's jar (needs Java 8 or above) and run this command:

$ java -jar bfg.jar  --replace-text replacements.txt  my-repo.git

The replacements.txt file should contain all the substitutions you want to do, in a format like this (one entry per line - note the comments shouldn't be included):

PASSWORD1 # Replace literal string 'PASSWORD1' with '***REMOVED***' (default)
PASSWORD2==>examplePass         # replace with 'examplePass' instead
PASSWORD3==>                    # replace with the empty string
regex:password=\w+==>password=  # Replace, using a regex

Your entire repository history will be scanned, and all non-binary files (under 1MB in size) will have the substitutions performed: any matching string (that isn't in your latest commit) will be replaced.

Full disclosure: I'm the author of the BFG Repo-Cleaner.

Roberto Tyley
  • 24,513
  • 11
  • 72
  • 101
2

I wrote a script for this a little while ago. You can find it here: https://gist.github.com/dound/76ea685c05c4a7895247457eb676fe69

(original writeup viewable from archive.org: https://web.archive.org/web/20160208235904/http://dound.com:80/2009/04/git-forever-remove-files-or-folders-from-history/)

The script builds on the git-filter-branch tool which comes with git. If you're curious, you can read more about removing files from a git repo here, but using the script from the link above should be easy and all you really need to accomplish removing that private information.

David Underhill
  • 15,896
  • 7
  • 53
  • 61
  • This looks like a great tool but I'm not sure it will work in my case. I should've been more clear in my question, but what I want to remove is the occurrence of certain strings - I don't want to delete entire files altogether because the files are still used. – Jimmy Feb 08 '10 at 23:18
  • Oh, I see. That is a bit trickier. If you entered these private strings in commits which don't contain anything else that you want to keep, then you can use git-filter-branch to remove just those commits (without deleting the files). My script can't do that for you, but if you check out the man page for git-filter-branch I think you'll see how you can use it to remove individual commits too. – David Underhill Feb 08 '10 at 23:21
  • Also, you should be able to use git-filter-history to apply a custom filter (script) over your files. This sounds like it might be a bit harder than simply removing a file or commit, but it should do what you want (and sounds better than restarting your repository and losing all your history when you release it to the public). – David Underhill Feb 08 '10 at 23:23
  • In the case that the text to substitute contains unusual characters (ie $, \, etc - for instance, with stronger passwords), it can be a bit fiddly getting the character escaping right for git-filter-branch. http://stackoverflow.com/questions/18647400/git-filter-branch-to-remove-strings-but-where-strings-contain-and-other-c is a question targeted at that particular problem. – Roberto Tyley Sep 11 '13 at 21:42
  • @DavidUnderhill the link above is now dead - may want to put it in a gist instead – Matt Vukomanovic Aug 09 '18 at 12:19
  • @MattVukomanovic Thanks for the heads up. Updated the answer to reflect this. – David Underhill Aug 20 '18 at 02:10