126

Question:
Is there a way to automatically checkout git submodules via the same method (ssh or https) as the main repository?

Background:

We have a non-public gitlab repository (main) that has a submodule (utils) which is also hosted as a non-public gitlab repository on the same server. Those repositories can be accessed either via ssh or https:

  • user@gitlabserver.com:my/path/repo.git
  • https://gitlabserver.com/my/path/repo.git

Both variants obviously require different forms of authentication and depending on the client computer and the user, one or the other is preferred.

For the top level repository (main) that is not an issue, as anyone can choose the method he or she prefers, but for the sub module this depends on the .gitmodules file and hence is (initially) the same for all.
Now instead of everyone having to adapt the .gitmodules file to whatever they prefer and make sure they don't accidentally commit those changes, it would be nice, if there was a way to just specify the server and repo path and git chooses either the same method that is used for the main repo, or something that can be set in gitconfig.

MikeMB
  • 20,029
  • 9
  • 57
  • 102
  • tldr sorry. but I use git-hooks to run `git submodule foreach git submodule update` or similar. I have no time to write a proper answer but i hope this will help. – Szabolcs Dombi Jun 19 '17 at 12:05

2 Answers2

151

I finally solved this problem by specifying the submodules url as a relative path:

So lets say your main git repository can be reached

  • either via https://gitlabserver.com/my/path/main.git
  • or via user@gitlabserver.com:my/path/main.git

And the .gitmodules file looks like this:

[submodule "utils"]     
    path = libs/utils   
    url = https://gitlabserver.com/my/path/utils.git

That would mean that even when you check out the main application via ssh, the submodule utils would still be accessed via https.

However, you can replace the absolute path with a relative one like this: 

[submodule "utils"]     
    path = libs/utils   
    url = ../utils.git

and from now on use

  • either git clone --recursive https://gitlabserver.com/my/path/main.git
  • or git clone --recursive user@gitlabserver.com:my/path/main.git

to get the whole repository structure which ever way you want. Obviously that doesn't work for cases where the relative ssh and the https paths are not the same, but at least for gitlab hosted repositories this is the case.

This is also handy if you (for whatever reason) mirror your repository structure at two different remote sites.

MikeMB
  • 20,029
  • 9
  • 57
  • 102
  • 1
    This also won't work if a module points to a different repository, right? For example if this is the central project https://username@stash.company.net/abc/win/win-agent.git and this one a referenced module https://username@stash.company.net/abc/mob/module.git – David Jun 03 '19 at 16:21
  • 1
    @David: I can't test it, but I don't see, why it wouldn't work (obviously you'd have to use `../../mob/module.git` in this case). – MikeMB Jun 04 '19 at 08:27
  • I tested it with ../ updating doesn't fail though but it doesn't seem logical for me, it shouldn't work, it should only work with ../../ as you wrote. – David Jun 04 '19 at 10:16
  • Ok, following worked: root project username@stash.company.net/scm/win/win-agent.git referencing module username@stash.company.net/scm/mob/module.git with following config: [submodule "modules/module"] path = modules/module url = ../../mob/module.git – David Jun 04 '19 at 11:46
  • 4
    Thank you for this, one additional thing I had to, after altering the `.gitmodules`, do was run; `git submodule sync` (reference; https://dhoeric.github.io/2017/https-to-ssh-in-gitmodules/) – Sander Looijenga Feb 25 '20 at 11:32
  • This is not a general solution, because the naming pattern of the ssh and https urls might not match. But in the case that was haunting the the last few days, this trick worked perfectly, and I bet it does quite widely, so THANKS! – lewis Feb 29 '20 at 05:15
  • 1
    I just used `git submodule add ../utils.git libs/utils` and it worked. No need to change if later. – Mitar Apr 14 '20 at 08:47
  • March 2021, and this does not work for me. Still no permission to clone the submodule, with relative path. It works flawlessly in my own terminal emulator. – Gauthier Mar 26 '21 at 13:54
  • This works well, but may cause some grief with **forks**. Since it's a relative path, git will look for a fork of each of the submodules as well, eg in `github.com/your-user/REPO`. Every developer who works on a fork will have to fork and maintain each submodule. – Akom Apr 29 '21 at 19:39
  • @Akom you can use `../../username/Repo` this way it will still use the repository from the original user in forks – maxbachmann Sep 08 '21 at 13:53
51

I think I have a more general answer, but it's not quite automatic. There's a configuration setting you can apply in ~/.gitconfig that will replace a particular URL with the one of your choosing.

For Github I have this in my ~/.gitconfig:

[url "ssh://git@github.com/"]
    insteadOf = https://github.com/

Here's a repository with a submodule declared with an HTTPS url that I can clone recursively using HTTPS.

git clone --recursive https://github.com/p2/OAuth2.git

OR

git clone --recursive git@github.com:p2/OAuth2.git

You can read about its use here: https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf

And here: https://git-scm.com/docs/git-clone

Breedly
  • 12,838
  • 13
  • 59
  • 83
  • Thanks. That setting looks quite useful. I' d still prefer to use relative paths where possible, because I can just tell someone else to recursively clone repository X without further manual configuration. However, it seems perfect from the other end: I want to clone someone else's repository tree that doesn't use relative paths with a different method (or mirror) than the author intended. – MikeMB Jun 28 '20 at 07:52
  • And if course you can't always work with relative paths. – MikeMB Jun 28 '20 at 07:54
  • Sure thing. I noticed this doesn't work super well with XCode and Swift package manager, but neither do relative submodules. – Breedly Jun 28 '20 at 18:21
  • Unfortunately this only seems to work with the argument passed to `git clone`. It doesn't work with the submodules themselves. – Timmmm Feb 14 '23 at 14:37