80

I'm trying to learn the new git-subtree command which was added in Git 1.7.11. I seem to lose ability to rebase after I add a subtree. I have the primary repository with README file and a library repository which also has a README file. I add it to lib directory with subtree add:

$ git subtree add -P lib/mylib myliborigin master

This works fine, but now the history looks like this:

*   22c1fe6 (HEAD, master) Merge commit 'b6e698d9f4985825efa06dfdd7bba8d2930cd40e' as 'lib/mylib' - 
|\                                                                                                                
| * b6e698d Squashed 'lib/mylib/' content from commit d7dbd3d
* b99d55b Add readme
* 020e372 Initial

Now when I want to rebase my repo against origin/master and it fails because the squash commit is applied directly against its parent commit which does not apply, because it is applied to the root of the repo and not the prefix I gave it to it when adding the subtree.

The reason for this is pretty clear if I look at the squash commit. There is no information about the prefix. It is just the original mylib commits squashed together. Only the next merge commit knows anything about it, but rebase does not take it to account here.

Are there any workarounds (besides never rebasing over the subtree commits)?

Anthony Mastrean
  • 21,850
  • 21
  • 110
  • 188
esamatti
  • 18,293
  • 11
  • 75
  • 82
  • 9
    Have you tried `git rebase --preserve-merges` ? – weynhamz Oct 12 '12 at 15:13
  • @TechliveZheng That worked perfectly, thanks! [Read more here](https://stackoverflow.com/questions/15915430/what-exactly-does-gits-rebase-preserve-merges-do-and-why) – helmesjo Jul 11 '17 at 08:33

9 Answers9

25

This works in simple cases:

git rebase --preserve-merges master

Thanks to @Techlive Zheng in the comments.

You may see

fatal: refusing to merge unrelated histories
Error redoing merge a95986e...

Which means that git failed to automatically apply your subtree. This puts you in the situation @ericpeters described in his answer. Solution:

Re-add your subtree (use the same command you originally used):

git subtree add -P lib lib-origin master

Continue the rebase:

git rebase --continue

And you're all set!

If you're wondering if it worked successfully, you can compare with your original version after rebasing to make sure you didn't change anything:

git diff <ref-before-rebase> <ref-after-rebase> -- .

(the -- . at the end instructs git to only diff the files in your current directory).

If all else fails and you don't care about preserving the commits themselves, you can simply git cherry-pick the original subtree commit.

The commit message will look something like Add 'lib/' from commit '9767e6...' -- that's the one you want.

Kevin Cooper
  • 5,018
  • 4
  • 37
  • 51
  • 1
    When you say "re-add", is the implication to also remove first? I am getting a `prefix already exists` message when doing the above `git subtree add` command. – Mike-E May 27 '18 at 04:33
  • 1
    @Mike-EEE in the situation described I believe git has failed to add it automatically, so by "re-add" I mean add it yourself manually. It should not exist at that point due to the rebase. – Kevin Cooper May 27 '18 at 18:13
  • Thanks for the context and the reply, Kevin. I have spent the day being bamboozled by this topic and my own personal plight, which derives from a seemingly innocuous `git subtree pull` on a [remote commit comprised of a single file change](https://github.com/SuperDotNet/Super.NET/commit/463724f3484774c7d6bc8afe6837eeb2045398a7) that results in dozens of conflicts and errors in the parent repo/branch that only differs from said commit. In any case, I was able to get what I was looking for here: http://git-memo.readthedocs.io/en/latest/subtree.html Perhaps it will assist someone else. – Mike-E May 27 '18 at 20:45
  • For the relatively simple case of "I forgot to pull some changes before adding the subtree and want to rebase", this works just as expected - thank you! – towo Jun 16 '18 at 11:39
  • 1
    A little late to the party, but this solution worked perfectly, thanks Kevin! I also got a `prefix already exists` message and it was because it contained a generated file (*.pyc) that is not version-controlled and prevented removal of the folder. Manually deleting the folder before doing `git subtree add ...` did the trick. Another note, if your original `git subtree add ... ` command had a `--squash`, remember to include it here also so that the rebased version will look exactly the same. – Jimmy He Dec 14 '20 at 07:28
  • 1
    Can confirm this worked too, I got the refusing to merge unrelated histories error. I had to use the rebase flags from @nyanpasu64's comment though. Otherwise I was getting a merge conflict with the README.md of my repo and the subtree. In hindsight I suppose it would have been easier to drop the subtree merge commits and then rebase and add them manually instead. But anyways: `git rebase --rebase-merges --strategy subtree master` And then following your steps to readd worked perfectly. – nhooyr Sep 10 '22 at 06:02
13

This is an old question, but I just had the same problem with my repo, and I finally found a complete solution, which (hopefully) preserves all the subtree metadata.

Suppose we had this commit tree:

B   (master) Add README.md
|     
A            Initial commit

and we forked a feature branch with a subtree residing in lib/:

git remote add -f githublib https://github.com/lib/lib.git
git subtree add --prefix lib/ githublib master --squash

It creates a merge commit D with two parents: our current master (B), and an unrelated commit F with the squashed history of the external repo. This commit also contains some git subtree metadata in its commit message (namely, git-subtree-dir and git-subtree-split).

   D     (feature) Merged commit 'F' as 'lib/'
  / \    
 /   F             Squashed 'lib/' content from GGGGGG
B        (master)  Add README.md
|     
A                  Initial commit

Later, we add some commits to both branches independently.

   E     (feature) Remove .gitignore from lib/
C  |     (master)  Add LICENSE.md
|  D               Merged commit 'F' as 'lib/'
| / \    
|/   F             Squashed 'lib/' content from GGGGGG
B                  Add README.md
|     
A                  Initial commit

Now we want to rebase feature onto master. Here's how:

1. Cherry-pick the commits from feature one by one to create a new copy of the feature branch on top of master.

git branch -f feature C
git checkout feature
git cherry-pick D E

E'       (feature) Remove .gitignore from lib/
|
D'                 Merged commit 'F' as 'lib/'
|
|  E               Remove .gitignore from lib/
C  |     (master)  Add LICENSE.md
|  D               Merged commit 'F' as 'lib/'
| / \    
|/   F             Squashed 'lib/' content from GGGGGG
B                  Add README.md
|     
A                  Initial commit

Now we have the equivalent of a rebase, but we've lost all information about the external repo, required for git subtree. To restore it:

2. Add the missing parent link as a graft, and rewrite the history of feature to make it permanent.

git checkout feature
git replace --graft D' C F
git filter-branch --tag-name-filter cat -- master..

And now we get a picture exactly equivalent to the one with started with. The old commits D and E are still out there, but they can be garbage-collected later.

E'       (feature) Remove .gitignore from lib/
|
D'                 Merged commit 'F' as 'lib/'
|\
| \                
C  \     (master)  Add LICENSE.md
|   \              
|    \   
|     F            Squashed 'lib/' content from GGGGGG
B                  Add README.md
|     
A                  Initial commit

Warning: This rewrites the history of feature, so be wary of publishing it if anybody else collaborates with you on this branch. However, since you wanted to make a rebase in the first place, you are probably aware of that :-)

Community
  • 1
  • 1
Skiminok
  • 2,801
  • 1
  • 24
  • 29
11

This isn't a solution, but its the current work around I use...

Using your initial example:

*   22c1fe6 (HEAD, master) Merge commit 'b6e698d9f4985825efa06dfdd7bba8d2930cd40e' as 'lib/mylib' - 
|\                                                                                                                
| * b6e698d Squashed 'lib/mylib/' content from commit d7dbd3d
* b99d55b Add readme
* 020e372 Initial

Rebase interactively to the 2nd commit before the subtree add:

$ git rebase -i 020e372

Delete the two subtree entries & mark edit for the prior commit:

e b99d55b Add readme

Save file/close, then when it gets to the "Add readme" commit, run the amend command:

$ git commit --amend

Then re-add your new subtree:

$ git subtree add -P lib/mylib myliborigin master

Continue the rebase:

$ git rebase --continue

Your branch should then be rebased off of master, and the subtree will be "normal" with the Squash + the Merge intact:

*   22c1fe6 (HEAD, master) Merge commit 'b6e698d9f4985825efa06dfdd7bba8d2930cd40e' as 'lib/mylib' - 
|\                                                                                                                
| * b6e698d Squashed 'lib/mylib/' content from commit d7dbd3d
ericpeters
  • 407
  • 5
  • 16
  • What if I have the scenario where I want to squash later commits made to the subproject that are showing up in the superproject's history. For example: commit 597df58 (HEAD, master) A commit to superproject commit 9cd82bd A commit to subproject commit 059ffbe A commit to subproject commit af5209e Merge commit 'da151a...' as 'subproject' commit da151a3 Squashed 'subproject/' content from commit 4d455fb commit b10f7c9 initial commit to superproject I want to squash 9cd82bd 059ffbe with af5209e because subproject's history has already been pushed to its own repo using git subtree push... – cowbert Sep 25 '14 at 05:22
  • not sure, I think I'd need a better look of what the project layout looked like, make sure to keep the terminology consistent, there are subtrees and there are submodules, but no such thing as a subproject :) – ericpeters Sep 25 '14 at 22:09
  • well the subproject is just the result of subtree add --prefix=subproject repoB branch – cowbert Sep 26 '14 at 03:28
  • Hopefully, you haven't destroyed the other repository because you thought everything would be fine later :( – Anthony Mastrean Sep 25 '15 at 15:52
  • This certainly works, but it's a bit hacky. See my answer [below](https://stackoverflow.com/a/48293315/763231) for a simpler method to achieve the same result. – Kevin Cooper Jan 17 '18 at 04:45
8

Git 2.24.0 (released 2019-11-04) added support for git rebase --rebase-merges --strategy [strategy]. So now if you run git rebase --rebase-merges --strategy subtree [branch] when your current branch contains a subtree merge, it will Just Work now.

For my project, I've decided I might not use git subtree add, but instead throw away the second parent of the merge commit using git replace --edit. I've also used the Git book v1's obsolete "subtree" tutorial, which does the same thing but is tedious.

nyanpasu64
  • 2,805
  • 2
  • 23
  • 31
  • 4
    In my first attempt using it, it does not "Just Work", at all. Git rebase blows up with `fatal: refusing to merge unrelated histories` if I try and rebase a branch with subtree commits on top of the latest master/main branch. – Tim Harper Mar 31 '22 at 22:29
  • This *sort of* worked for me - it didn't error but I ended up with an ancient version of the subtree - I think possibly because it moves all the subtree-merges to the start of the rebase then merges the earliest non-subtree?. I ended up just `rebase -i`ing and dropping all the subtree squash-merge commits, then rebasing, then remerging the subtree. Not applicable in all situations but ended up being a lot simpler to get my head around. – markson edwardson Jan 01 '23 at 23:12
4

Apparently this is expected behaviour (for some perverse definition of "expected behaviour.") See: https://web.archive.org/web/20200219001959/http://git.661346.n2.nabble.com/subtree-merges-lose-prefix-after-rebase-td7332850.html.

Not that this is much help to anyone. I'd love to find a workaround for this too.

kirbyfan64sos
  • 10,377
  • 6
  • 54
  • 75
Nick Hutchinson
  • 5,044
  • 5
  • 33
  • 34
  • I really feel like this anti-feature needs to be advertised more clearly. The behavior is completely perplexing and everyone should know about it going in to subtrees, and yet, the documentation does not highlight it. It took me a while to find this stackoverflow article. – Tim Harper Mar 31 '22 at 22:18
  • Above link is broken now. I believe https://marc.info/?l=git&m=133062667420408 is identical to what was linked. – nhooyr May 25 '22 at 19:18
2

I had a similar issue: I wanted to rebase after doing a subtree add, and using --preserve-merges still left me with a merge conflict (due to conflicting .gitignore files and others).

In my case, I didn't necessarily plan on using any subtree functionality: I was simply pulling in a repo that should have been part of the superproject originally. In case it helps anyone else, here's what I ended up doing, based off of other related answers I found.

Suppose I have two projects, main_project and sub_project, in the same directory. I want to pull sub_project into a directory named sub_project within main_project, assuming neither repo has ever had a directory named sub_project:

cd main_project
git fetch ../sub_project
git checkout -b sub_project FETCH_HEAD
git filter-branch --prune-empty --tree-filter '
    if [[ ! -e sub_project ]]; then
        mkdir -p sub_project
        git ls-tree --name-only $GIT_COMMIT | xargs -I files mv files sub_project
    fi'
git checkout branch-to-merge-within
git merge sub_project
git branch -d sub_project

I'll update if I find any problems with this approach.

Community
  • 1
  • 1
David Alan Hjelle
  • 942
  • 1
  • 10
  • 23
2

You can edit the interactive-rebase contents manually to re-add the subtree from the original squashed commit, using the exec command, without stopping and doing any manual work.

Start the rebase by using --rebase-merges:

git rebase -i --rebase-merges <commit>

  • Commit graph at the start:

    *   22c1fe6 (HEAD, master) Merge commit 'b6e698d9f4985825efa06dfdd7bba8d2930cd40e' as 'lib/mylib' - 
|\                                                                                                                
| * b6e698d Squashed 'lib/mylib/' content from commit d7dbd3d
* b99d55b Add readme
* 020e372 Initial

  • Auto generated interactive rebase contents:

    reset [new root]
pick b6e698d Squashed 'lib/mylib/' content from commit d7dbd3d 
label b6e698d9f4985825efa06dfdd7bba8d2930cd40e-2

reset onto
pick 020e372 Initial
pick b99d55b Add readme
merge -C 22c1fe6 b6e698d9f4985825efa06dfdd7bba8d2930cd40e-2 # Merge commit 'b6e698d9f4985825efa06dfdd7bba8d2930cd40e' as 'lib/mylib'

  • New contents:

    reset [new root]
pick b6e698d Squashed 'lib/mylib/' content from commit d7dbd3d 
label b6e698d9f4985825efa06dfdd7bba8d2930cd40e-2

reset onto
pick 020e372 Initial
pick b99d55b Add readme
# We have to remove the subtree folder first, otherwise `git subtree` will complain about its existence.
exec rm -r lib/mylib
# Re-add the subtree, but from the existing commit.
# NOTE: `--squash` is important, otherwise a duplicate squash commit will be created.
exec git subtree add --prefix=lib/mylib b6e698d --squash
# We can remove the last `merge` action as we are manually adding the subtree

Caveats

If you've pulled or merged after adding the subtree you will have to apply this procedure to those commits too but changing the command to the corresponding one (merge/pull/etc).

bergercookie
  • 2,542
  • 1
  • 30
  • 38
Gilberto Valadez
  • 71
  • 6
0

You need to use

git rebase --preserve-merges --preserve-committer --onto new_place start end
Adam Dymitruk
  • 124,556
  • 26
  • 146
  • 141
0

An alternative way is to git rebase -i master, then substitute the merged squashed commit with b (break here), so git rebase would stop at the time of the original subtree add commit. Run git subtree add ... again at that moment, then git rebase --continue. You would get the same history but rebased onto the latest master.

lz96
  • 2,816
  • 2
  • 28
  • 46