20

I have a project with a handful of submodules. Many of them are cloned from a GitHub fork to which I've added a branch for my custom mods. A typical setup is like thus:

In local folder: MyProject1/Frameworks/SomeAmazingRepo/

$ git branch -vva
*my-fork                       123456 [my-fork/my-fork] Latest commit msg from fork
master                         abcdef [origin/master] Latest commit msg from original repo
remotes/my-fork/my-fork        123456 [my-fork/my-fork] Latest commit msg from fork
remotes/my-fork/master         abcdef [origin/master] Latest commit msg from original repo
remotes/origin/HEAD            -> origin/master
remotes/origin/master          abcdef [origin/master] Latest commit msg from original repo

$ git remote -v
my-fork                        git@github.com:MyUser/SomeAmazingRepo.git (fetch)
my-fork                        git@github.com:MyUser/SomeAmazingRepo.git (push)
origin                         git://github.com/OriginalOwner/SomeAmazingRepo.git (fetch)
origin                         git://github.com/OriginalOwner/SomeAmazingRepo.git (push)

I git clone --recursive my project to begin a new spin-off project and when it begins to recurse, it spits out an error claiming it can't find the stored commits for these repos. Upon inspection it seems that the remotes haven't been added and the branch is left (empty) in master ...

In local folder: MyProject2/Frameworks/SomeAmazingRepo/

$ git branch -vva
*master                        abcdef [origin/master] Latest commit msg from original repo
remotes/origin/HEAD            -> origin/master
remotes/origin/master          abcdef [origin/master] Latest commit msg from original repo

$ git remote -v
origin                         git://github.com/OriginalOwner/SomeAmazingRepo.git (fetch)
origin                         git://github.com/OriginalOwner/SomeAmazingRepo.git (push)

The only remedy is to go and add the remotes manually to all the repos (very tedious).

There exists a similar issue in the cases where there are two tracking branches as above but only one remote (origin => my github fork). In these case, it finds the commit and checks it out but fails to recreate the tracking branch, leaving a "dangling" commit...very scary as it doesn't warn you!

How do I clone my project so that it reliably recreates the submodules' remotes and branches?

Hari Honor
  • 8,677
  • 8
  • 51
  • 54
  • Couldn't you first make sure all the trackinf branch are in place, first for your parent repo, then through a [`git submodule foreach`](http://stackoverflow.com/a/1032653/6309), for each submodule, using a one-liner [like this one](http://stackoverflow.com/a/6300386/6309). – VonC Jul 11 '12 at 19:35
  • What do you mean "trackinf branch"? Also, I don't necessarily want all remotes checked out as branches. The problem is that `git clone --recursive` is not recreating the submodules' (non origin) remotes at all. I feel I'm missing some nuance here but I can't quite put my finger on it... – Hari Honor Jul 12 '12 at 09:05
  • Sorry, it was a typo: "tracking branches" (http://stackoverflow.com/questions/4693588/git-what-is-a-tracking-branch). Apparently not what you are missing though. – VonC Jul 12 '12 at 09:29
  • 1
    Actually, the second scenario which I briefly mention at the end is when the submodule in the source repo has only one remote, master (which clones fine), but a tracking branch is checked out. In this case `git clone--recursive` sets HEAD to the branch's commit (which it successfully finds because it was part of remote=origin) but fails to recreate the branch itself meaning it's dangling. Perhaps it's a different problem but my question I guess is simply 'is git clone --recursive broken or is it just me?!?' or 'why is cloning with submodules so unreliable and how do I work around it?' – Hari Honor Jul 12 '12 at 10:57
  • Ok, regarding the "unreliability", I think I have a good explanation for it. See the last part of my answer below. – VonC Jul 12 '12 at 12:21

1 Answers1

36

git clone --recursive is equivalent to git submodule update --init --recursive.

And a git submodule update will only checkout the recorded SHA1 (recorded in the parent repo):

Update the registered submodules, i.e. clone missing submodules and checkout the commit specified in the index of the containing repository.
This will make the submodules HEAD be detached.

2012: So finding no active branch in a submodule is the norm.
A git submodule foreach 'git checkout master' can at least set the master branch (if you are sure that all the recorded SHA1 were supposed to be part of a 'master' branch for each submodules.

2013-2014: you can configure your .gitmodules file in order to specify a branch to checkout in your submodule.
See "How do I update my git submodules from specific branches?"

cd /path/to/your/parent/repo
git config -f .gitmodules submodule.<path>.branch <branch>

Any remote that you add locally in a submodule, like my-fork, aren't recorded in the parent repo at all.
So when you clone again that parent repo, it will initialize and update the submodules as recorded in the .gitmodules file (you can change that address, but only one is associated with each submodules).
If you have other remote address to associate to each submodule, you need a script to automate the process.

As explained in "True nature of submodule", a submodule is primarily there to record/access a fixed point in the history.
You can develop directly within a submodule, but you need to go there and make the right branch and/or add the right remotes.

it spits out an error claiming it can't find the stored commits for these repos.

Every time you make a commit in a submodule, you need to:

  • push it to the associated remote (ie, the one recorded in the .gitmodules of the parent repo)
  • go back to the parent repo and commit said parent.

But:

If you have pushed to 'my-fork' while the associated remote repo of that submodule was not 'my-fork'... then the next clone won't be able to checkout that submodule commit.

Update August 2014 (Git 2.1)

See commit 9393ae7 by Matthew Chen (charlesmchen):

submodule: document "sync --recursive"

The "git submodule sync" command supports the --recursive flag, but the documentation does not mention this.
That flag is useful, for example when a remote is changed in a submodule of a submodule.

Update Git 2.23 (Q3 2019)

You can also consider git clone --recurse-submodule --remote-submodules: that will clone with submodules already checked out at their tracking branch instead of their gitlink parent repo SHA1.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • Right. I think I'm starting to get it. The problem occurs because I've originally added my submodule from RepoA and then forked it to Github RepoB and then *added that as a new remote in the submodule*, which ParentRepo knows nothing about. Instead I should update ParentRepo's(?) `.gitmodules` to set the *remote/origin* to RepoB and run `git submodule sync` (link #3) to get everything in order? – Hari Honor Jul 12 '12 at 13:40
  • And for scenario #2, I just have to ensure that my submodules are living in branch=master and run `git submodule foreach...` **or** manually go and check everything out. Seems like a bit of a loophole that ParentRepo can't work out what branch it's submodules are on. Ideally one would want one command that completely clones the state of a repo, submodules and all. In the first scenario I can see the rationale - I've set features in the submodule which I haven't told the parent about - but as far as branches, all the info is contained in the submodule's source repo....feature request? – Hari Honor Jul 12 '12 at 13:45
  • Thx by the way for the great answer. I hope this saves someone else some headache. – Hari Honor Jul 12 '12 at 13:46
  • @HariKaramSingh I am glad you made it work. I confirm for your `git submodule sync` usage. As for the branches, don't forget they can be renamed/deleted at any time. The commit SHA1 is what is important. The "configuration" build with submodules is a collection of commits, not, like with "svn externals", the "latest versions of some branches": git submodules and svn externals are fundamentally different (and not compatible): http://stackoverflow.com/questions/3131912/why-are-git-submodules-incompatible-with-svn-externals/3132221#3132221 – VonC Jul 12 '12 at 15:15