14

I'd like to use git submodule.

The steps I need to take to push my changes to my project are

  1. add/commit/push from submodule directory
  2. add/commit/push from parent directory

Steps I need to take to pull changes of my project.

  1. git pull from parent directory
  2. git submodule update from parent directory

Steps for updating the submodule from its original repo

  1. git pull from submodule directory

What worries me is the following exerpt from http://git-scm.com/book/en/Git-Tools-Submodules

The issue is that you generally don’t want to work in a detached HEAD environment, because it’s easy to lose changes. If you do an initial submodule update, commit in that submodule directory without creating a branch to work in, and then run git submodule update again from the superproject without committing in the meantime,(?? update/commit/update will lose change?) Git will overwrite your changes without telling you. Technically you won’t lose the work, but you won’t have a branch pointing to it, so it will be somewhat difficult to retrieve.

To avoid this issue, create a branch when you work in a submodule directory with git checkout -b work or something equivalent. When you do the submodule update a second time, it will still revert your work, but at least you have a pointer to get back to.

I'm going to modify submodules and don't wanna mess up, the doc above briefly mentions the possibility of losing change, and I don't understand what might cause the loss.

I wonder what additional steps more than I listed above I need to take to prevent the loss. Especially several team members modify submodules, what do they need to do not to mess up?

eugene
  • 39,839
  • 68
  • 255
  • 489
  • yes of course if you modify a submodule without commit/push, when you make submodule update, it will checkout the origin submodule head, so you will left your changes, to not have this situation, you should create a branch, modify submodule&commit, if you make submodule update, you will leave your branch to origin submodule head, don't worry, you can now merge your submodule branch or just checkout to have it. – elhadi dp ıpɐɥןǝ Jan 09 '13 at 11:23
  • can you be more specific? suppose two programmers are modifying the submodule for a team project. what do they have to do? each of them create a branch with same name/modify/commit/push in submodule folder. and submodule update, then? – eugene Jan 10 '13 at 02:23
  • I'm trying to prepare a more in depth answer for you now as I'm working on something similar. In the meanwhile, have you read http://blog.endpoint.com/2010/04/git-submodule-workflow.html and http://stackoverflow.com/questions/5814319/git-submodule-push? The first link also covers creating a branch with the submodule (if you want to understand why read 'Issues with Submodules' here http://git-scm.com/book/en/Git-Tools-Submodules). – Aaron Newton Jan 10 '13 at 11:43
  • Can I also suggest you check out this post - http://stackoverflow.com/questions/6500524/git-subtree-or-gitslave-if-switch-away-from-git-submodules. There are several approaches to this, and I'm increasingly looking at git-slave. – Aaron Newton Jan 10 '13 at 12:43
  • I yet have to investigate how to avoid this problem, but I understand what they mean: if you run `git submodule update` then always a detached HEAD checkout of the submodule is performed. This is normal behavior. If subsequently you commit changes to that detached HEAD you won't be able to find back those commits after another `git submodules update` which would again reset the detached HEAD branch to *before* those commits. – Carlo Wood Dec 22 '16 at 14:41
  • I now solved my problem by having created the git submodule https://github.com/CarloWood/cwm4 that seamless integrates (other) git submodules into my autotools supported projects. – Carlo Wood Jan 06 '17 at 00:06

1 Answers1

25

I'd like to share my experiences with you as someone who works with external projects in Visual Studio solutions trying to solve a similar problem. I'm relatively new to git, so if anyone has any constructive criticism I would appreciate that.

If you're using Visual Studio, the Git Source Control Provider extension is free (http://visualstudiogallery.msdn.microsoft.com/63a7e40d-4d71-4fbb-a23b-d262124b8f4c), and seemed to recursively commit submodules when I tested it out.

HOWEVER I'm using VS Web Developer Express at home for development, so I don't want to rely on an extension (I also think it's good to have some idea of what's happening under the hood). Therefore I've been forced to figure out the commands, and I've added some notes below.

NOTES

If you haven't done already, have a thorough read through http://git-scm.com/book/en/Git-Tools-Submodules. There are quite a few caveats, and I will refer back to this page. If you try to work with submodules without reading this, you will give yourself a headache very quickly.

My approach follows this tutorial, with a few added extras: http://blog.endpoint.com/2010/04/git-submodule-workflow.html

Once you have your superproject initialised (e.g. git init && git remote add origin ...), start adding your submodules like so:

git submodule add git://github.com/you/extension1.git extension
git submodule init
git submodule update

Check that your .gitmodules file reflects this addition, e.g.

[submodule "extension1"]
        path = extension
        url = git://github.com/you/extension1.git

Switch to your submodule directory (i.e. cd extension). Run:

git fetch #I use fetch here - maybe you can use pull?
git checkout -b somebranchname #See the Git-Tools-Submodules link above for an explanation of why you need to branch

I made a change here to README.txt so I could commit it (also so I would have a record of what I was doing in this commit), then commited the module to apply the branch (still within the submodule directory):

git add .
git commit -a -m "Branching for extension submodule"

Now go into the superproject (i.e. cd ..). You will also need to commit here (if you look at the git submodule page I mentioned it explains why this is necessary):

git status #will show you that your submodule has been modified
git commit -a -m "Commiting submodule changes from superproject"

Now we can recusively push our projects like so if required:

git push --recurse-submodules=on-demand

You will need to run through the above steps once for all your submodules.

Once you have done this for all your submodules and started making changes you want to commit and push, you can use:

git submodule foreach 'git add .' #recursively add files in submodules

Unfortunaly I haven't found a way to recursively commit without using something like git-slave (anyone?), so you then need to go into each submodule directory and run a regular commit for the files you just added. In the superproject:

git status #tells you that `extension` submodule has been modified
cd extension
git commit -a -m "Commiting extension changes in superproject edit session"

Once the submodule is commiting, you'll also need to commit the superproject (again), so:

cd ..
git add .
git commit -a -m "Altered extension submodule"
git status #should now show 'working directory clean', otherwise commit other submodules in the same manner

This can get slightly annoying (because you end up committing twice), but once you're aware of it it's actually not so bad (as it forces you to check what you're committing in each project). Just my opinion - if you've isolated some of your superproject's functionality into submodules, it should work in isolation from the rest of your projects anyway (so commiting them at different times while annoying is not the end of the world).

Now we can push once more...

git push --recurse-submodules=on-demand

If you then descend into your submodule and try and push again, you'll find it won't do anything as the latest commit has already been pushed.

Cloning (or using a remote origin) for a superproject can also be quite confusing - such as the need to run git submodule update twice after git submodule init. Read the 'Cloning a Project with Submodules' section of http://git-scm.com/book/en/Git-Tools-Submodules.

Something that caught me out when cloning my superproject was getting the latest changes for the submodules. See Easy way pull latest of all submodules

My variant of this is to use a 'development' branch for checked out submodules (but you can call it whatever you want) and then use this in the superproject:

git submodule foreach git pull origin development

When I set this up I also swap to the branch I want to push my changes to on the checked out submodule like so:

cd extension
git checkout -b development #This will tell you this is a new branch, but I believe this means a new branch of the local git repository - this will get pushed to the 'development' branch
#Make your changes, commit etc.

I can confirm that when I follow the above steps, changes to the submodules in a clone/remote origin project (when pushed) showed up in other clones/remote origins of the same project (not forgetting that last submodule pull command).

I hope that was of some use to you.

Community
  • 1
  • 1
Aaron Newton
  • 2,124
  • 1
  • 28
  • 31
  • Thanks for detailed answer, I decided I should put off this submodule thing for a moment.. – eugene Feb 02 '13 at 01:09
  • I understand. I'm a bit of a Ubuntu junkie so I feel quite at home with the out-of-the-box BASH shell that comes with the Windows installer for Git and have been working with that. If you are going to be using it with some sort of IDE project setup (e.g. as I described for Visual Studio) it is probably worth looking into a decent plugin for your IDE of choice. – Aaron Newton Feb 02 '13 at 08:04
  • 4
    [To commit for each submodule all at once](http://stackoverflow.com/questions/19728933/continue-looping-over-submodules-with-the-git-submodule-foreach-command-after) you can use: `git submodule foreach 'git commit -a -m "Altered submodule" || :'` – Cel Jun 21 '16 at 08:19
  • 1
    I don't feel this really answers the question (I have this question too). It certainly doesn't explain why the documentation warns about losing commits, nor does it show how to prevent co-workers to screw up like that. Surely, if you follow the workflow you describe, nothing will get lost, but then one wouldn't be screwing up in the first place. I think a correct answer would explain what script and hooks will prevent said screw up. – Carlo Wood Dec 22 '16 at 14:11
  • I have a repository containing submodules, I use this command `git submodule update --init --recursive --remote --merge path/to/submodule`. Then I run `git add`, `git commit -m` and `git push`. Now other devs who have cloned my repository recursively, then run `git pull`, but the change to the submodule appears as an unstaged change after pulling, what's the right way to pull from an upstream repository with changed submodule. – CMCDragonkai May 12 '17 at 06:54
  • This is symptomatic of a merge conflict - have you checked for this? I've started using the SourceTree GUI since I wrote this post. I can see that it pulls using three separate commands - one for the fetch, one for the pull and one for the submodule update. Fetch: `git -c diff.mnemonicprefix=false -c core.quotepath=false fetch origin`, pull: `git -c diff.mnemonicprefix=false -c core.quotepath=false pull origin authentication` (for a branch named `authentication`) and submodule update: `git -c diff.mnemonicprefix=false -c core.quotepath=false submodule update --init --recursive`. – Aaron Newton May 14 '17 at 12:54