How to completely remove a file from git if you already pushed it to the remote branch, merged to the develop branch and it is not the latest commit

Question

I've seen a lot of tutorials explaining how to do this in different scenarios but seems like many of them talk of the latest commit. So what I need is to remove this sensitive file completely from both branches: feature-branch and develop.

How do I do that?

I've found this recipe:

git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch path_to_file" HEAD

Will it work in my case? Namely, will it completely remove this file from all the branches?

EDIT:

I've decided to use BFG and here's the output it has given me after I run

bfg --delete-files 'filename_of_the_file_to_delete'

output:

Found 273 objects to protect
Found 198 commit-pointing refs : HEAD, refs/heads/develop, refs/heads/develop-with-relative-paths, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 7d97ab00 (protected by 'HEAD') - contains 1 dirty file :
    - src/email/filename_of_the_file_to_delete (16.1 KB)

WARNING: The dirty content above may be removed from other commits, but as
the *protected* commits still use it, it will STILL exist in your repository.

Details of protected dirty content have been recorded here :

/Users/albert/Documents/projects/rjx/rjxfp/.git.bfg-report/2020-08-25/21-50-38/protected-dirt/

If you *really* want this content gone, make a manual commit that removes it, and then run the BFG on a fresh copy of your repo.


Cleaning
--------

Found 486 commits
Cleaning commits:       100% (486/486)
Cleaning commits completed in 699 ms.

Updating 7 Refs
---------------

    Ref                                                    Before     After
    --------------------------------------------------------------------------
    refs/heads/develop                                   | e9c3c4ba | 53c5dd39
    refs/heads/feature-icons-for-top-level-cats          | d7dde80c | 377ae820
    refs/heads/feature-user-profile                      | 7d97ab00 | e3b1b336
    refs/remotes/origin/develop                          | e9c3c4ba | 53c5dd39
    refs/remotes/origin/feature-icons-for-top-level-cats | d7dde80c | 377ae820
    refs/remotes/origin/feature-user-profile             | 7d97ab00 | e3b1b336
    refs/stash                                           | 9fc9a356 | 39945789

Updating references:    100% (7/7)
...Ref update completed in 54 ms.

Commit Tree-Dirt History
------------------------

    Earliest                                              Latest
    |                                                          |
    ..........................................................DD

    D = dirty commits (file tree fixed)
    m = modified commits (commit message or parents changed)
    . = clean commits (no changes to file tree)

                            Before     After
    -------------------------------------------
    First modified commit | 6a211a6b | 35597e71
    Last dirty commit     | e9c3c4ba | 53c5dd39

Deleted files
-------------

    Filename                          Git id
    ------------------------------------------------------
    filename_of_the_file_to_delete    | 4121d724 (16.1 KB)


In total, 37 object ids were changed. Full details are logged here:

    /Users/albert/Documents/projects/rjx/rjxfp/.git.bfg-report/2020-08-25/21-50-38

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive

The file I wanted to delete is still in the directory where it was created, at least on the branch from which I ran the bfg command. Then I don't think I completely understand what it says about some protected dirty content. Who protected it and why? It says: If you *really* want this content gone, make a manual commit that removes it, and then run the BFG on a fresh copy of your repo.

I don't understand what I should do exactly.

As far as I understand, the commit it mentioned 7d97ab00 (which is now e3b1b336) is the last commit on the branch from which I ran the command, so I have to remove the file (but do I have to remove it with rm or git-rm?) then make a commit and run BFG again?

Does this answer your question? [Git: Remove committed file after push](https://stackoverflow.com/questions/18357511/git-remove-committed-file-after-push) — joshmeranda, Aug 25 '20 at 18:16
@puffin As far as I understand, this answer is applicable if we speak of the latest commit and like I said I made a merge, so this tutorial is not my case. — Albert, Aug 25 '20 at 18:21
The simplest solution is to rollback the merge into `develop`, follow these instructions for `feature-branch`, and redo the merge after the sensitive info is removed — joshmeranda, Aug 25 '20 at 18:24
Does this answer your question? [How to remove file from Git history?](https://stackoverflow.com/questions/43762338/how-to-remove-file-from-git-history) I recommend the [BFG repo cleaner](https://rtyley.github.io/bfg-repo-cleaner/). — Thomas, Aug 25 '20 at 18:25
Its not recommended to use filter branch since its slow and tends to mangle your history, see the [`filter-branch`](https://git-scm.com/docs/git-filter-branch) docs. It suggests using [`filter-repo`](https://github.com/newren/git-filter-repo/) — joshmeranda, Aug 25 '20 at 18:28
@puffin Would you please write the whole command? Will it be enough to change `filter-branch` to `filter-repo`? There're not too many commits in the repository, maybe about 500 commits. — Albert, Aug 25 '20 at 18:31
@Albert, no you'll need to install a git extension. You can read more [here](https://www.mankier.com/1/git-filter-repo) `git filter-repo --filename-callback 'if filename == : return None'` — joshmeranda, Aug 25 '20 at 19:01
@Thomas I've decided to use `BFG` but I need to clarify something, would you please have a look? — Albert, Aug 25 '20 at 19:13
There are branches , where you want to keep the file? E.g. master? — Christoph, Aug 25 '20 at 19:30
@Christoph No, I want it to be removed entirely from the project, that's to say, not a single branch should have it. — Albert, Aug 25 '20 at 19:32
@Christoph I've made an update in the question. Would you please have a look? — Albert, Aug 25 '20 at 19:37
Whats the difference to the duplicate then? Delete the file, commit and remove it. — Christoph, Aug 25 '20 at 19:37
@Thomas Yes, I've managed to remove the file off the project, thanks for reminding me of BFG, nice tool. The case was not a trivial one. — Albert, Aug 25 '20 at 20:09

torek · Accepted Answer · 2020-08-26T00:28:08.493

While I haven't used The BFG, I understand from its documentation that it considers the tip-most commit of each branch to be a "correct state". That is, suppose you want to remove the file secret.txt from every commit. If you use The BFG with the instruction "remove file secret.txt", it will remove it from all commits except the current commit (and any other branch-tip commit that has that file in it).

Remember that branch names simply identify some commit, by its commit hash ID. It's the commit itself that has the files in it. Every commit has a full snapshot of every file. So if you added secret.txt four commits ago, and have this:

... <-H <-I <-J <-K <-L   <--master

where each uppercase letter stands in for a commit, the file secret.txt is in commits L, K, J, and I. It's not in H because H is five commits ago.

The backwards-pointing arrows here are how Git works: every commit leads backwards to the previous commit. No part of any commit can ever be changed (not by The BFG either), so what The BFG must do is to create new and improved commits, then throw out the old commits entirely.

The BFG will copy existing commit I to a new-and-improved I' in which secret.txt does not exist. It will then copy J to a new-and-improved J' in which secret.txt does not exist, and repeat this for K. But L is the last commit, identified by name, so The BFG assumes that you mean to keep secret.txt there, because it's there now and identified by name. This is the "protected" commit. So The BFG copies L to L'—it has to do this because it copied K to K' and the existing L points back to the existing K—but this time it keeps secret.txt in commit L'.

You end up with:

... <-H [ XXX deleted: <-I <-J <-K <-L ]
       \
        I' <-J' <-K' <-L'   <-- master

in which secret.txt now exists only in that last commit, L', that was protected.

The documentation for The BFG suggests that you do this:

git rm secret.txt
git commit

before you start, so that you start with:

... <-H <-I <-J <-K <-L <-M   <--master

where new commit M doesn't have secret.txt in it. Now commits I through L can all be fixed-up, because L isn't the last commit any more. It is not identified by name. The name master finds, not L, but M; only M itself finds L.

Notes

Once you've updated your own repository to have new-and-improved commits, and have thrown out the old bad ones, you will need to use git push --force to get any other Git that still has, and is still using, the old bad commits, to switch to the new and improved commits.

Always assume that if secrets.txt was available on the web for even a few seconds, someone out there grabbed a copy of it.

Wow, grand explanation! Thank you a lot! – Albert Aug 25 '20 at 23:19 — Albert, Aug 25 '20 at 23:19

How to completely remove a file from git if you already pushed it to the remote branch, merged to the develop branch and it is not the latest commit

1 Answers1

Notes