Files added to .gitignore lost after attempting to squash a series of commits through interactive rebasing

Question

I've been adding some files to .gitignore over a few commits, that I do not want pushed to remote. This includes files such as .Rprofile, which may contain information I do not want shared, and *.code-workspace, which I did not see as necessary to track. After committing everything was working as intended, but then I decided to squash the .gitignore changes into one commit using git rebase -i. I did this in terminal in VS Code, which caused to program to crash. Long story short, after that all files listed in .gitignore are gone! Not just the files recently added to gitignore, but also previous files that has been in gitignore since I initiated git. This includes .Renviron.

I therefore have three questions:

What happened? Why did git tamper with files in the ignore list? Isn't that the whole point of an ignore list: so that git will not tamper with them?
Are the ignored files lost, or is there a way they can be retrieved? At this point I consider it best if there a way to restore the working directory to its state prior to attempting the rebase.
How do I prevent this from happening again? These were files that was deliberately ignored because I need them in my working directory, but do not want Git to push them to the remote repository. However, I certainly do not want git to delete them!

Full history:

1. Initial git log output:

% git log

commit 7df4f... (HEAD -> master, origin/master)
Author: 
Date:   Tue Aug 22 10:42:52 2023 +0200

    New commit.

commit 43c8c...
Author: 
Date:   Tue Aug 22 10:32:39 2023 +0200

    Ignoring code testing notebook.

commit 7a4be...
Author: 
Date:   Fri Aug 18 15:46:28 2023 +0200

    Ignoring VS Code workspace

commit 0866e...
Author: 
Date:   Fri Aug 18 15:39:46 2023 +0200

    Ignoring .Rprofile

commit c3b51...
Author: 
Date:   Fri Aug 18 15:38:27 2023 +0200

    .gitignore ignoring .Rprofile

commit 299bd
Author: 
Date:   Fri Aug 18 13:36:33 2023 +0200

    `bugs()` output directory ignored

commit 36699
Author: 
Date:   Fri Aug 18 13:34:35 2023 +0200

    Old commit.

commit ...

2. Attempting interactive rebasing:

% git rebase -i HEAD~7

pick 36699 Old commit.
pick 299bd `bugs()` output directory ignored
s c3b51 .gitignore ignoring .Rprofile
s 0866e Ignoring .Rprofile
s 7a4be Ignoring VS Code workspace
s 43c8c Ignoring code testing notebook.
pick 7df4f New commit.

Essentially I wanted to squash commits 299bd - 43c8c together. When trying to save the rebase file, VS Code crashed and at this point all files listed in .gitignore were gone.

3. `git log` output in Console:

% git log

commit 3cbe9... (HEAD)
Author: 
Date:   Fri Aug 18 13:36:33 2023 +0200

    # This is a combination of 4 commits.
    # This is the 1st commit message:
    
    `bugs()` output directory ignored
    
    # This is the commit message #2:
    
    .gitignore ignoring .Rprofile
    
    # This is the commit message #3:
    
    Ignoring .Rpofile
    
    # This is the commit message #4:
    
    Ignoring VS Code workspace

commit 36699...
Author: 
Date:   Fri Aug 18 13:34:35 2023 +0200

    Old commit.

Apparently, the latest commit is "lost". Not unexpected, git branch showed that I'm in the middle of a commit.

% git branch
* (no branch, rebasing master)
  master

4. Attempting to continue rebase:

git rebase --continue 
fatal: could not read log file '.git/rebase-merge/message': No such file or directory
error: could not commit staged changes.

5. Switched to master branch:

% git checkout master
Warning: you are leaving 1 commit behind, not connected to
any of your branches:

  3cbe9 # This is a combination of 4 commits. # This is the 1st commit message:

If you want to keep it by creating a new branch, this may be a good time
to do so with:

 git branch <new-branch-name> 3cbe9

Listing the files in the directory ls -a showed that the ignored files are still absent.

git statusprovided the following output:

% git status

On branch master
Your branch is up to date with 'origin/master'.

Last commands done (6 commands done):
   squash 7a4be Ignoring VS Code workspace
   squash 43c8c Ignoring code testing notebook.
  (see more in file .git/rebase-merge/done)
Next command to do (1 remaining command):
   pick 7df4f New commit.
  (use "git rebase --edit-todo" to view and edit)
You are currently editing a commit while rebasing branch 'master' on '499cb'.
  (use "git commit --amend" to amend the current commit)
  (use "git rebase --continue" once you are satisfied with your changes)

nothing to commit, working tree clean

6. Moved commit "3cbe9" to a new branch:

..given the warning message when checking out master.

% git branch apprentice 3cbe9

7. I then attempted to continue the rebase:

% git rebase --continue
The previous cherry-pick is now empty, possibly due to conflict resolution.
If you wish to commit it anyway, use:

    git commit --allow-empty

Otherwise, please use 'git rebase --skip'
On branch master
Your branch is up to date with 'origin/master'.

Last commands done (7 commands done):
   squash 43c8c Ignoring code testing notebook.
   pick 7df4f New commit.
  (see more in file .git/rebase-merge/done)
No commands remaining.
You are currently rebasing branch 'master' on '499cba5'.
  (all conflicts fixed: run "git rebase --continue")

nothing to commit, working tree clean
Could not apply 7df4f... New commit.

git log output:

% git log
commit 7df4f (HEAD -> master, origin/master)
Author: 
Date:   Tue Aug 22 10:42:52 2023 +0200

    New commit.

commit 43c8c
Author: 
Date:   Tue Aug 22 10:32:39 2023 +0200

    Ignoring code testing notebook.

commit 7a4be
Author: 
Date:   Fri Aug 18 15:46:28 2023 +0200

    Ignoring VS Code workspace

commit 0866e
Author: 
Date:   Fri Aug 18 15:39:46 2023 +0200

    Ignoring .Rpofile

commit c3b51
Author: 
Date:   Fri Aug 18 15:38:27 2023 +0200

    .gitignore ignoring .Rprofile

commit 299bd
Author: 
Date:   Fri Aug 18 13:36:33 2023 +0200

    `bugs()` output directory ignored

commit 7df4f (HEAD -> master, origin/master)
Author: 
Date:   Tue Aug 22 10:42:52 2023 +0200

    New commit.

commit 43c8c
Author: 
Date:   Tue Aug 22 10:32:39 2023 +0200

    Ignoring code testing notebook.

commit 7a4be
Author: 
Date:   Fri Aug 18 15:46:28 2023 +0200

    Ignoring VS Code workspace

commit 0866e
Author: 
Date:   Fri Aug 18 15:39:46 2023 +0200

    Ignoring .Rpofile

commit c3b51c
Author: 
Date:   Fri Aug 18 15:38:27 2023 +0200

    .gitignore ignoring .Rprofile

For some reason the commit history has now been duplicated, while commit 36699 ("Old commit.") is seemingly gone. The files in the ignore list remains absent.

Where the files _added/deleted_ in previous commits (the ones you are trying to squash)? Say... it *might* have been like this: you add/commit add file blah, then realize the file should be ignored, so you remove it (keeping it in the working tree), add it to `gitignore`, commit..... then you decide you want to squash these 2 commits so that the act of adding the file and then deleting it are not visible in the history. Is that how it happened? Then depending on how delicate you were about handling the file, it might end up being deleted after squashing. — eftshift0, Aug 22 '23 at 19:52
Following the previous scenario, and so that it is clear why that happens, what should git do if you checkout the commit where you _added_ the file? It might warn you that your file in the working tree (untracked/ignored) is different from what is in the commit.... but that is only if the file is different.... so say, the file is the same, git sees no problem with checking it out... so... it goes through.... and now, you want to go back and checkout the tip of the branch. What should git do about the file _now_? There is a _removal_ of the file in place... git **has** to remove the file. — eftshift0, Aug 22 '23 at 19:55
You should have done `git rebase --abort` instead of `git checkout master` or any of the following stuff. — hobbs, Aug 22 '23 at 20:02
@eftshift0 Some of the files were added in previous commits and then removed from repository but kept in the working directory using `git rm --cached my.file`. I suspected this may have caused issues. This is not the case for all deleted files, however. For example, the `.Renviron` was added to `.gitignore` prior to the first commit to the repository, and therefore never neither added nor subsequently deleted. However, it was still removed during the attempted squash. — Pål Bjartan, Aug 22 '23 at 21:23
@hobbs Hindsight is 20/20. This would have been the wiser course of action, but was not done. I need to know if there's a way to salvage the files / return to the previous state, given the current circumstances. — Pål Bjartan, Aug 22 '23 at 21:27
One comment regarding `I've been adding some files to .gitignore over a few commits, that I do not want pushed to remote.`. This is perfectly fine (and I do the same all the time), however you should have some version history discipline and make such [**temporary commits**](https://stackoverflow.com/a/75010635/23118) properly stand out. I typically have a `==== .gitignore ====` commit where I ignore log files or other crap I have lying around in my repo to not pollute `git status` etc which I then filter out before pushing the branch for pull request. — hlovdal, Aug 22 '23 at 22:24
Regarding "lost" files, git does not immediately trash commits the moment they are no longer referenced by anything (I think the default cleanup/garbage collect period is something like a month), and you are able to view all commits you have had active with `git reflog` or `gitk --reflog`, which includes non-purged no longer reachable commits. `git reflog -- .gitignore` should list all recent commits modifying .gitignore, and then you can create new branches to bring those commits back. — hlovdal, Aug 22 '23 at 22:36
..... Or get specific files from those commits with `git checkout some-commit-id -- some-file` or `git restore --source=some-commit-id -- some-file` — eftshift0, Aug 22 '23 at 22:39
Can you check what entries you have in your stash : `git stash list` ? and see if some of these entries contain some of the files you are looking for ? — LeGEC, Aug 23 '23 at 08:46
also: look into vscode own file history: if you don't have a `.Renviron` file anymore, try re-creating an empty one, and see if the "Timeline" section of vscode shows traces of the previous content for that file — LeGEC, Aug 23 '23 at 08:50
@eftshift0 I think you are right the deletion was most likely caused by them being added/deleted in the commit history. Turns out. `.Renviron` was always located in my home directory (sorry for the confusion), so I am now fairly certain the rebase/squash only affected files that were initially added and subsequently deleted throughout the commit history. Good news, I was able to recover by going through the files in each of the last commits, using `git checkout some-commit-id -- some-file`. — Pål Bjartan, Aug 23 '23 at 11:02
@hlovdal I'm not entirely certain what you mean by "temporary" commits in my particular case, as I just wanted to correct an error, namely prevent certain files from being pushed to remote. That being said, this is unarguably a good practice in many cases, and something I will definitely use in the future! =) ..and thankf for the tips for how to use `git reflog`. — Pål Bjartan, Aug 23 '23 at 11:08
Maybe the phrase *private* rather than *temporary* works better in some cases. — hlovdal, Aug 23 '23 at 11:36
@hlovdal Just to be clear: By *temporary*/*private*, do you mean these are commits that you add temporarily your local repo, but remove before pushing to remote? I could see how that makes sense. With regards to my issue above, wouldn't deleting a commit, where a file is removed from cache, replay this action and I risk losing the file again? — Pål Bjartan, Aug 24 '23 at 09:22
`commits that you add temporarily your local repo, but remove before pushing to remote` Yes, exactly! With regards to keeping such commits for later usage, assuming your current branch is `myfeature`, create an extra branch (`git branch mylocalsetup myfeature`) first, then cleanup the feature branch (`git rebase -i main`). After that rebase your local changes on top of the cleaned up branch (`git rebase myfeature mylocalsetup`). After `myfeature` is merged into `main` then rebase `mylocalsetup` on top of that (`git rebase main mylocalsetup`). — hlovdal, Aug 24 '23 at 11:06
Then for your next feature branch you can make use of `mylocalsetup` and then use that as a base instead of `main` (e.g. `git checkout -b myfeature2 mylocalsetup`). How you structure the work flow is a bit of personal preference, alternative 1: strictly keep all local changes in `mylocalsetup` and when new changes are required, you check out that branch, do modifcations and then rebase `myfeature2` on top (`git rebase --onto mylocalsetup $THE_SHA_OF_MYLOCALSETUP_BEFORE_LATEST_CHANGE myfeature2`). Cleanup operation then becomes `git rebase --onto main mylocalsetup myfeature2`. — hlovdal, Aug 24 '23 at 11:22
Alternative 2: when local changes are needed, interleave those commits among the normal commits on `myfeature2`. Cleanup operation: `git checkout mylocalsetup; git merge myfeature2` and then interactive rebase and rebases as I described for `myfeature` in previous comment. — hlovdal, Aug 24 '23 at 11:23

minorChaos · Accepted Answer · 2023-08-24T02:50:07.063

1

Warning: I think you are still in the 'rebase' mode and all steps you do might count towards your rebasing activity.

I will try to collect what I understand with the suggestions of the comments. Please correct me if anything is wrong.

May be that, as you say, ''the commit history has now been duplicated'' because log sometimes shows two histories mixed. Use git log --graph --decorate or gitk to see something that we can interpret. The other possible reason is that during your interrupted rebase, you are allowed to add any number of commits (an example of that is described in SPLITTING COMMITS section of man git-rebase) and this is what you may have unintentionally done by git checkout master followed by git rebase --continue.

I think it is the second because I was able to do something similar (though I had to add some git add and/or git commit -- but I had no crash here.)

Remedy:

Note the original commit 7df4f, as backup. (May be git branch goodmaster 7df4f). (Actually I see you have it recorded as origin/master, too.) Also backup all files, because you may have added to .gitignore some more than you already lost.
finish rebase by git rebase --abort. You will have to do it sometime in future anyway. And whatever it does to your files, it is better to suffer now. (And shorter in-rebase-history also means less damage.)
You may now look at the state by gitk, or look what might be recovered by gitk --reflog. Look at git status.
Now I would recover the git repo to some reasonable state -- especially if it is not. This might be git reset 7df4f or git reset --hard 7df4f where the later also changes the working tree (i.e., the files). The later hence can delete files for which you have done git rm (when playing with .gitignore). May be git checkout master works the same but maybe not.
recover missing files by git checkout some-commit-id -- some-path-and-file. You can find some commit ids by gitk --reflog. Also git log -- filename can reveal the commits where you added/deleted the file. Or git reflog -- filename.

.gitignore This file tells git setup not to list specified files. But no more than that.

What happenend? I think you have done git rm --cached files when you were adding them to .gitignore. Even if not, you possibly have done at least git add files for them some time ago. This way, in the history there are records of this files being created/deleted. If you make git to move you through the history, they have to appear/disappear accordingly. (File .gitignore makes no influence.)

How to avoid this situation Well. Something like that happened to me too. May be to convince the git developers to warn you before doing such a thing both 1. as an immediate result of a command 2. as a temporary action in course of something like rebase. Hmm, well. ... Sounds complicated, but there might be a chance for implementation...(?)

edited Aug 24 '23 at 02:50

answered Aug 24 '23 at 02:42

minorChaos

101
7

Thanks for the elaborate answer. I don't think I was able to abort rebase, as I already attempted to continue. Turns out I had part of the commit history and the successful squash on the branch created in OP's step 6, but lacking the last. On master, I did recover the files the way you desribe in step 5 and added to a new commit. – Pål Bjartan Aug 24 '23 at 09:12
(continued) Due to a lot of trial and error, my master branch was getting cluttered, so instead of trying to fix this one, I rebased the last two commits on top of 'apprentice' (`git rebase -i HEAD~2 --onto apprentice`) and overwrote master with apprentice's commit history (`git reset --hard apprentice`) and force-pushed to remote. Anyway, I think your answer addresses my questions 1 and 2. Regarding question 3: There really should be a way to let ignored files escape the commit history altogether. – Pål Bjartan Aug 24 '23 at 09:13
1

@PB for `rebase --continue` there are conditions to satisfy, and `rebase ` may get interrupted again. `rebase --continue` is conditional by design. Comparing to that, `rebase --abort` is meant is one-off action and is designed to finish always successfully (There might be fatal cases, of course.) – minorChaos Aug 24 '23 at 09:19
@PB (continued) ignored files: When walking through the history, and there are file adds/deletes, walking *should* mean adding and deleting the file. But may be conflict with the content of .gitignore could result in a warning. – minorChaos Aug 24 '23 at 09:22
I agree, I think understand better now why the files "disappeared". ..and yes, a warning would certainly have been nice. With regards to my comment on question 3: I see one problem with how adding a previously committed file to `.gitignore`: `git rm --cached files` doesn't seem to affect previous commits. This may be desirable in many cases, but let's say I by accident added a file to the repo that contained sensitive information (e.g. `.Renviron`), which is subsequently pushed to a public remote? – Pål Bjartan Aug 24 '23 at 09:38
(continued) Several commits later, I realize my mistake and add the sensitive file to .gitignore. While `git rm --cached files` will remove it from any commits going forward, it can still be accessed in the commit history. It makes me think if there is (or should be) a way to remove a file from the commit history altogether, like it never existed..? – Pål Bjartan Aug 24 '23 at 09:42
"Revriting the history is a bad idea.", "Publishing rebased branches is a bad idea". Both are true and actually parallel. The second is explained in `git help rebase`. – minorChaos Aug 24 '23 at 10:05
When you want to rewrite the history (remove a sensitive file), it is what rebase is designed for. Suppose that commit `abac` was the one created by `git add sensitive; git commit`. I think the following should work (of course, with clean tree and clean index, no unfinished `rebase` et cetera). `git rebase -i abac`, and delete the first line with `pick abac`. – minorChaos Aug 24 '23 at 10:13
1

And if commit `abac` contained some desirable actions instead of adding the sensitive file, try `git rebase -i abac` with replacin `pick abac` by `edit abac` a then ammend the commit by `git rm sensitive; git commit --amend; git rebase --continue`. (Is that correct?) – minorChaos Aug 24 '23 at 10:17