0

I have a code directory, that is currently in svn, which contains multiple sub-directories:

/code
├ Dockerfile        # create images for CI containing these packages
├───clitools/       # command line interface to the other repos (qa make-docs, qa test-package, etc.)
├───doclib/         # code for creating/collecting/building docs
├───glapi/          # gitlab api code
├───pkgtools/       # code for working with our standard package structure
├───repolib/        # common svn/git/hg interface
├───synclib/        # fetch code from multiple repos with heterogenous VCSs
├───runners/        # code to run sync/async/parallel tasks
└───testpackage/    # linters, analyzers, testers, builders

these subdirectories are Python packages and live in their individual git repositories. They are svn:ignored (we're using gitlab so we can't list them as svn:externals) and our tooling can fetch latest from all of them with one command.

Some of them import and use code from the others, so it is not uncommon to work on features that span multiple repos.

There is a chicken-and-egg problem whenever a new feature (e.g. when we added the possibility of mapping an s3 bucket into our dev-tree) is added to our global syncspec, where the command to fetch latest from all 100+ repositories:

qa fetch-global ./syncspec.txt

first needs to pull the latest from all code/* repos and re-run itself with the new syncspec.

There is also an issue when bootstrapping a new developer machine/server, where all the code/* packages have to be synced and installed before we can fetch all. We pip install -e . all packages to make development faster.

If everything was on svn we could list the sub-packages as svn:externals and checkins at the code/ level would pick up all changes in the sub-packages - and svn up code would pull everything. I've read about git submodules, but everyone seems to think they are a bad idea.

Is there a (preferably simple) way to set things up in git so that git clone/pull/push on the code repo would update everything? I.e. with changes in code/doclib and code/glapi I want git push code to commit to both repos - without having to go to those ("upstream") repos to merge my changes?

Maybe I'm a little bit stuck in a svn mindset and should be approaching this differently?

Currently I'm thinking I'll either need to put all the packages into a giant code repo containing subdirectories for each package, or use tooling to gloss over the fact that these are separate functionalities but should most of the time be treated as a single unit.

thebjorn
  • 26,297
  • 11
  • 96
  • 138
  • 1
    "I've read about git submodules, but everyone seems to think they are a bad idea" So you know the Git way to do this, but you just don't want to? – matt Feb 17 '23 at 12:03
  • Android/AOSP [uses repo](https://source.android.com/docs/setup/create/repo) for this purpose. I'm don't know of anyone else that uses it, but maybe it is what you're looking for. [Josh](https://github.com/josh-project/josh) is another alternative approach (that I've [read about on here](https://stackoverflow.com/a/75408776/40342)). – Joachim Sauer Feb 17 '23 at 12:08
  • @matt lolz, maybe, but everything I've read about git submodules seems to indicate that they're more geared towards creating your own little island and committing to the "origin" repos is "challenging" - not to mention all the specialized commands needed. git subtrees seems interesting as well, but I can't say I understand the differences between submodules and subtrees (but committing to "origin" repos seem challenging with subtrees as well). I don't expect git to be as easy as subversion, but this is _very_ easy in svn and currently pretty opaque for me in git... – thebjorn Feb 17 '23 at 13:02
  • See "[Differences between git submodule and subtree](https://stackoverflow.com/a/31770147/6309)" and "[Why are git submodules incompatible with svn externals?](https://stackoverflow.com/a/3132221/6309)". For a all repository, submodule remains the best option. – VonC Feb 17 '23 at 13:21
  • @VonC but our main project does not "depend on a fixed version of another component (repo)", quite the opposite, we want to track latest of all "components". – thebjorn Feb 17 '23 at 13:27
  • 1
    @thebjorn then "[git submodule tracking latest](https://stackoverflow.com/a/9189815/6309)" – VonC Feb 17 '23 at 13:31
  • 1
    The decision between git and SVN should be quite simple (and while I'm a huge fan of git, that's not where I'm going..). If the decision seems hard, perhaps the technology itself should be questioned; is an SCM the right tool for the job at all? ;) – micromoses Feb 17 '23 at 14:14

1 Answers1

1

After commenting, while submodule is not a good equivalent for all what a svn:external link can do, the OP wants to track latest of all "components".

For that, you can declare a submodule which follows a branch

# add submodule to track master branch
git submodule add -b master [URL to Git repo];

# update your submodule
git submodule update --remote

That is one way to record multiple "component" states in a parent repository.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250