2

My situation: I have a large number of computers that I use for various tasks. I have a large number of libraries, each in its own git repo.

My desire: I want to be able to modify one of the libraries on any computer, do a git commit/push; then go to another computer, do a git pull, and have all the libraries updated. Then I modify one of these libraries, commit/push, and everything works when I get tot he next computer.

My current attempt: I have a top-level git project that incorporates all the other library repos as submodules. This includes a .gitmodules file that specifies the working branch of each module by using

git config -f .gitmodules submodule.modulename.branch develop

I have update = merge set for each module. I have submodule.recurse set to true, so git pull at the top level does something to each module.

How it is broken: Heads become detached. I wrote a script that parses the .gitmodules file and does a checkout of the appropriate branch of each module. I then commit and push the top module. Whenever I modify things and try to do a pull, e.g. on another machine, heads become detached. If I don't notice that the head is detached before I start modifications, I have to carefully unscramble the wreckage before I can commit my changes.

There are literally 3.6k stack overflow questions about git detached heads over the past decade, and most seem to be from the submodule capability. I haven't gone through all of them, but what I have tried isn't working.

I forget why I rejected git-subtree, but git-subrepo hasn't been touched in over a year and has 153 issues and 25 pull requests pending, so I think it's dead.

Does anyone have a working solution to this?

Accepted answer from @vonC looks good.

I could probably streamline this a bit, but my readme for the top-level project now says:

Recommended checkout:

git clone --recursive --jobs=8 *mysuperproject_clone_url*
cd *mysuperproject*
git config alias.pullall 'submodule foreach git pull'
git config alias.statusall 'submodule foreach git status'
git config alias.switchall \
    "submodule foreach --recursive 'git switch \$(git config -f \${toplevel}/.gitmodules submodule.\${sm_path}.branch)'"
git switchall

Updating from repository

git pullall

If heads become detached, fix with

git switchall

To add a module

Module named newmodule working on path develop in the following example.

cd /path/to/mysuperproject
git submodule add git@github.com:myaccount/newmodule
git config -f .gitmodules submodule.newmodule.branch develop
git config -f .gitmodules submodule.newmodule.update merge

If the submodule is on the default master branch, you still have to config the branch.

If you switch a submodule to a different branch, then in the top level you must config again

git config -f .gitmodules submodule.newmodule.branch newbranch

And push both the submodule and the top level project.

On a different working directory (such as on a different machine), you must

cd /path/to/mysuperproject
git pull
git switchall
git pullall
DMPalmer
  • 163
  • 7
  • There is an ongoing (but probably far-future-dated) project for reworking the "user experience" (UX) for submodules, but this really is how they're intended to be used: as detached-HEAD setups virtually all the time. The general idea is to avoid using branch names at all in the submodules, since branch names have negative functional value at that point. – torek Jan 08 '22 at 04:38
  • What I mean by this is straightforward: a *branch name* represents the *latest* commit. A *submodule* in Git represents a *specific* commit. If that commit hash ID were to change just because some branch name somewhere changed, that would be incorrect behavior. So if a submodule is left "on" a branch, this is to be considered a deficiency. It may be a temporarily-necessary deficiency, but from the design-of-submodules point of view, it's a bug. – torek Jan 08 '22 at 04:40
  • @torek so you are saying that submodules is not the appropriate tool for my (seemingly common) use-case. The useful behavior is the incorrect behavior. So the question is: is there a correct tool in git? Or can the available tool be perverted to do the job. – DMPalmer Jan 08 '22 at 16:50
  • As far as I know, there is no correct tool. The Google folks wrote `repo` (the Python thing) to tackle this but seem to have abandoned `repo`, so apparently there's something deeply wrong with that too. – torek Jan 08 '22 at 20:32

1 Answers1

2

I mentioned before that git submodule update --remote --merge is supposed to not detached the HEAD of a submodule following a branch.

I understand you have set update = merge, but just for testing, try the complete update command, to see if this works.

Since the HEAD is still detached, you need to add (to a git alias script for instance) the command

git submodule foreach --recursive git switch $(git config -f .gitmodules submodule.${sm_path}.branch)

I just tested it:

First, in the Git repository, I check the submodule is in a detached HEAD mode:

vonc@vclp MINGW64 ~/git/git (master)
$ git submodule update --init
Submodule 'sha1collisiondetection' (https://github.com/cr-marcstevens/sha1collisiondetection.git) registered for path 'sha1collisiondetection'
Cloning into 'C:/Users/vonc/git/git/sha1collisiondetection'...
Submodule path 'sha1collisiondetection': checked out '855827c583bc30645ba427885caa40c5b81764d2'


vonc@vclp MINGW64 ~/git/git/sha1collisiondetection (master)
$ git br
* (HEAD detached at 855827c)
  master

Then I define my alias, with escaped $: \$.
No need for ../ when accessing the .gitmodules file. That is what $toplevel is for.

vonc@vclp MINGW64 ~/git/git (master)
$ git config alias.switchall \
    "submodule foreach --recursive 'git switch \$(git config -f \${toplevel}/.gitmodules submodule.\${sm_path}.branch)'"

Final test:

vonc@vclp MINGW64 ~/git/git (master)
$ git switchall
Entering 'sha1collisiondetection'
Previous HEAD position was 855827c Detect endianess on HP-UX
Switched to branch 'master'
Your branch is up to date with 'origin/master'.

vonc@vclp MINGW64 ~/git/git (tmp)
$ cd sha1collisiondetection/

vonc@vclp MINGW64 ~/git/git/sha1collisiondetection (master)
$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

vonc@vclp MINGW64 ~/git/git/sha1collisiondetection (master)
$ git branch
* master
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • Answers in edit to question – DMPalmer Jan 08 '22 at 01:16
  • @DMPalmer OK. I have edited the answer to propose a solution. – VonC Jan 08 '22 at 01:21
  • How and when would I do this? Can I add a 'hook' in git so that every time I do anything it re-attaches all heads? – DMPalmer Jan 08 '22 at 02:18
  • @DMPalmer You can define a git alias function which chains the two commands: https://stackoverflow.com/a/7005698/6309 – VonC Jan 08 '22 at 02:20
  • it isn't working for me, either because the ${sm_path} is being interpreted by my bash shell before git even starts, or because .gitmodules is not in the submodule's directory, or both. The second can be fixed by using ../gitmodules, but I haven't foudn the right pattern of escapes to fix the second. Alias: `switchall = submodule foreach --recursive git switch $(git config -f ../.gitmodules submodule.${sm_path}.branch)` looks in `submodule..branch` each time. – DMPalmer Jan 08 '22 at 18:21
  • Also, .git/config local to the top-level directory (as opposed to global '~/.git/config'), where I store the aliases doesn't seem to be something that can be placed under version control. Should I just have those alias commands in my setup instructions or is there a better way of doing it? – DMPalmer Jan 08 '22 at 18:24
  • @DMPalmer I just tested that alias. It does seem to work just fine. I have edited my answer to illustrate the steps I followed. – VonC Jan 08 '22 at 21:44