How do I deal with an error in my .gitmodules history?

Question

I get an error when I try to push a bit repo to a new server:

$ git push -f research master
Enumerating objects: 13060, done.
Counting objects: 100% (13060/13060), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3397/3397), done.
Writing objects: 100% (13060/13060), 5.59 MiB | 364.00 KiB/s, done.
Total 13060 (delta 9516), reused 13060 (delta 9516)
remote: Resolving deltas: 100% (9516/9516), done.
remote: error: bad config line 14 in blob .gitmodules
remote: error: object 4f098d01fb9671a7f5ff5f617ef89b289a8ea9bb: gitmodulesParse: could not parse gitmodules blob
remote: fatal: fsck error in pack objects
error: remote unpack failed: index-pack abnormal exit
To gitlab.research.com:proj/repo.git
 ! [remote rejected] master -> master (unpacker error)
error: failed to push some refs to 'git@gitlab.research.com:proj/repo.git'
$ 

Strangely, I get the same error when I run git fsck locally:

$ git fsck
Checking object directories: 100% (256/256), done.
warning in blob 4f098d01fb9671a7f5ff5f617ef89b289a8ea9bb: gitmodulesParse: could not parse gitmodules blob
Checking objects: 100% (13060/13060), done.
$ 

It turns out that "line 14" in the .gitmodules file is not in the current version of the file, which is only 11 lines long. The problem results because there was a version of .gitmodules that was committed which had a conflict.

Can I just delete the blob and push it to the new repo, or is there a deep problem that I cannot easily fix?

EDIT - NEW INFORMATION!

Given @torek's excellent posting below, combined with the script in Which commit has this blob? (the first script worked, but the second one did not), I was able to find the problematic commit:

$ ~/.bin/git-find-object 4f098d01fb9671a7f5ff5f617ef89b289a8ea9bb
06407fd merged changes from master and stats

My git repo only has one branch visible:

$ git branch -v
* master e8ff018 minor

I can check out the problematic commit:

$ git checkout 06407fd
Checking out files: 100% (321/321), done.
fatal: bad config line 14 in file /home/xv32/gits/repo/.gitmodules

$

I edited the .gitmodules file directly, and removed the offending lines:

<<<<<<< HEAD
        branch = ee
=======
>>>>>>> master

I did an git add .gitmodules

I did a:

$ git commit --amend -m 'fixed conflict in .gitmodules'

OMG it changed a lot of stuff.

I then did a git status:

$ git status
On branch master
Your branch and 'origin/master' have diverged,
and have 1 and 1 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   submodule1 (new commits)
        modified:   submodule2 (new commits)
        modified:   submodule3(new commits)

no changes added to commit (use "git add" and/or "git commit -a")

$

git fsck no longer reports problems:

$ git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (15206/15206), done.

But I am still unable to push to the remote repo:

$ git push -f research master
Counting objects: 13245, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3395/3395), done.
Writing objects: 100% (13245/13245), 5.68 MiB | 2.81 MiB/s, done.
Total 13245 (delta 9746), reused 13180 (delta 9702)
remote: error: bad config line 14 in blob .gitmodules
remote: error: object 4f098d01fb9671a7f5ff5f617ef89b289a8ea9bb: gitmodulesParse: could not parse gitmodules blob
remote: fatal: fsck error in packed object
error: remote unpack failed: index-pack abnormal exit
To https://gitlab.research.com/repo/repo.git
 ! [remote rejected] master -> master (unpacker error)
error: failed to push some refs to 'https://gitlab.research.com/censyn-sensitive/das_decennial.git'

$

Also, when I do the git --amend, a whole bunch of important things disappear, beyond the bad lines in the object.

Honestly, I do not understand why I am even getting an error, since the error is not at the top of the tree.

Answer 1

Can I just delete the blob ...

No: there's a tree object T that says "I should have blob 4f098d01fb9671a7f5ff5f617ef89b289a8ea9bb". You must therefore find the tree and deal with that. Tree T is referred-to by some commit object C. Since the file is .gitmodules, this is probably the top-level tree of that commit. You might have to find that commit, but:

The problem results because there was a version of .gitmodules that was committed which had a conflict.

... this is the commit, or maybe just the first commit of several, that has a tree that says "I should have the blob". In other words, you've already found the commit in question.

The trick is now to rewrite that commit with a new-and-improved version. The new version should be very nearly the same as the original, except that its committed .gitmodules (a blob) should be one that is correct instead of one that is incorrect.

Once you replace that commit, you have to replace that commit's child or children, because the child/children refer to the bad commit by its hash ID, and the new, corrected replacement has a different hash ID. Of course, as soon as you replace those, you have to replace their children, and so on all the way down to any descendant branch tip commits.

Ideally you'd be able to just use git rebase to fix this (as phd suggested). Unfortunately, you said:

... had a conflict

which implies that the first bad commit is itself a merge. You cannot rebase a merge: you must instead construct a new, corrected merge.

Exactly how hard or easy this is depends on a number of things, such as how familiar you are with Git's inner workings and how many commits there are "downstream" of the bad merge. If there are only a few, you might want to just repeat the merge, this time without the bug, and then copy the one or two remaining commits by hand.

Suppose the bad merge is commit X, and that its first parent is commit P:

...--o--P--X--1--2   <-- master (HEAD)
          /
...---o--o   <-- otherbranch

There are just two follow-on commits here, so you could:

git checkout <hash-of-P>
git merge otherbranch

The same conflicts will occur as last time, but this time you can resolve them correctly so that .gitmodules is correct. Then:

git cherry-pick <hash-of-1>
git cherry-pick <hash-of-2>

will copy commits 1 and 2 atop this new replacement merge:

          ,-X'-1'-2'  <-- HEAD
         / /
...--o--P-/-X--1--2   <-- master
         | /
         |/
...---o--o   <-- otherbranch

where X', 1', and 2' are the new-and-improved versions. (You might have to fix a merge conflict when cherry-picking whichever of 1 or 2 fixed .gitmodules, by taking the corrected .gitmodules; or Git might see that the two changes result in the same thing, and decide there's no conflict.)

Then you can force the name master to identify commit 1' instead, after which you can forget that commits X, 1, and 2 ever existed:

...--o--P--X'-1'-2'  <-- master (HEAD)
          /
...---o--o   <-- otherbranch

and now you can git push -f research master.

Alternatives

Besides this method, you can:

• git checkout the bad merge directly (git checkout hash-of-X);
• edit the .gitmodules file to fix it, and git add the result;
• run git commit --amend: this makes X'.

This is probably at least a little easier since you don't have to repeat the rest of the merge. You can then repeat the cherry-picking.

If there are a lot of commits "after" the bad merge, instead of cherry-picking them one at a time, you can use git rebase --onto to copy them en masse. This only works for sure if there are no merges in this series of commits. That is, the above kinds of graphs work fine, but:

                   3--4--5
                  /       \
...--o--P--X--1--2------6--7--8   <-- master (HEAD)
          /
...---o--o   <-- otherbranch

may not, because commit 7 is a merge: git rebase will drop it entirely and try to make a new chain atop X' that goes 1-2-3-4-5-6-8 or 1-2-6-3-4-5-8. That could be OK, depending on your needs.

The newest versions of Git have a --rebase-merges flag that will re-perform the merge (while copying the other commits a la git cherry-pick). You can use that for such cases.

Or, last, you can use git replace combined with git filter-branch. That's most appropriate if you have tens of thousands of commits "downstream" of the bad one and/or many branch-and-merge operations downstream. Otherwise you might as well stick with fixing the bad merge, then rebasing or cherry-picking the remaining commits.