18

I am working on doing a migration from SVN to Git. I have already used git-svn to get the history into a single git repository, and I already know how to use git-subtree to split that repository into smaller ones. This question is not about how to do the migration, it is about when to split and when not to split.

I want to split the large repository because some of the directories are self-contained libraries that are also shared with other projects. Previously an svn checkout was done on the library without the need to checkout the entire project. During all of this I discovered that there are probably dozens of directories that make sense to be in their own repository because they are 1) independent and 2) shared across projects.

Once you get above a handful of git repositories, it seems prudent to use a tool that makes working with many repositories easier. Some examples are Google's repo, git submodules, git subtree, and creating a custom script (it appears that chromium does this). I have explored these various methods, and understand how to use them.

So the question is about direction for the transition from subversion.

Should I try and stick to one large git repository, only splitting it into smaller pieces when absolutely necessary or should I split it into dozens or potentially hundreds of smaller repositories? Which would be easier to work with? Is there another solution that I have missed? If going with the many repositories, which tool should I use? What factors will make someone favor one method over another?

Note: The source needs to be checked out on Windows, MacOS, and Linux.

onionjake
  • 3,905
  • 27
  • 46
  • 1
    "What criteria should be used to determine whether to split a git repository?" Perhaps this is a better way to ask my question? – onionjake Mar 07 '14 at 21:38
  • Why do you need to use scripts to control subtree? Are you talking about automatically syncing them up between projects using hooks? Because, I use subtrees a lot and I purposefully do NOT want them syncing automatically. Often it is the case that a super-project that needs to use one of the libraries is left untouched for a long time, and when I come back to that project it's nice to find it still in working condition. If I continue on the project again later, I may then manually decide to pull in library changes, and handle any breaking changes in the super project. – johnb003 Mar 12 '14 at 02:24
  • @johnb003 a lot of the libraries are being actively developed and would need frequent updates. – onionjake Mar 17 '14 at 15:43
  • Relevant: http://programmers.stackexchange.com/questions/161293/choosing-between-single-or-multiple-projects-in-a-git-repository – onionjake Jul 21 '14 at 17:44

5 Answers5

6

That process can be guided by a component approach, where you identified coherent set of files (an application, a project, a library)

In term of history (in a source control tool), a coherent set means it will be labelled, branched or merged as a all, independently of the other set of files.

For a distributed version control system (like git), each of those set of files is a good candidate for a git repo of its own, and you can then group those you need for a specific project in a parent repo with submodules.

I describe this approach for instance in;

The opposite (keeping everything in one repo) is called "system-based approach", but can lead to huge Git repo, which, as I mentioned in "Performance for Git", isn't compatible with how Git is implemented.

The OP onionjake asks in the comments:

Could you please include more information on the subtleties of identifying components?

This process (of identifying "components", which in turn become git repos) is guide by the software architecture of your system.
Any subset which acts as an independent set of file is a good candidate for its own repo. It can be a library, or dll, but also part of an application (a GUI, a client vs. a server, a dispatcher, ...)

Each time you identify a group of tightly linked files (meaning modifying one will likely have effect to others), there should be part of the component, or in git, the same repo.

Community
  • 1
  • 1
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • This is an excellent answer, and left me wanting more. Could you please include more information on the subtleties of identifying components? Perhaps a detailed example of how it would be done? – onionjake Mar 10 '14 at 17:10
3

Personally I like small repos - they work well when you have a good dependancy management system like Composer for PHP.

It takes the pain away managing the check out process and also tracks versions etc.

It also permits repos to be hosted by different providers. We use a combination of bespoke code and open source repos.

BillyBigPotatoes
  • 1,330
  • 11
  • 23
  • +1 for mentioning composer, which appears to solve exactly this problem. Can it be used on non PHP projects? – onionjake Feb 21 '14 at 17:54
  • It is a php solution but alternatives exist for other languages e.g. maven for Java – BillyBigPotatoes Feb 21 '14 at 17:57
  • For python, there's [setuptools](https://pypi.python.org/pypi/setuptools). However, in general, these tools have little to do with criteria used to split projects apart. – johnb003 Mar 12 '14 at 02:16
2

I would say, go with subtrees most of time if not all the time - and feel free to make subtrees freely as you see necessary.

With lots and lots of dependencies, submodules start to become painful. If you have any effect on the development of those dependancies, then that goes doubly so. Submodule might be ok if you have a completely 3rd party library that doesn't change versions very often, and that you would never actively develop for as part of your larger project.

Submodules are too separated from the super-repo for dependencies you actually work on.

Example: If you make a change to a submodule, you have to commit on the submodule, push up, cd up to the super repo, add the submodule to the index/stage, commit it, and push up again. its a hassle of a workflow. Not to mention the hassle of removing, moving, or renaming a submodule.

Git subtrees are much better. The histories are intertwined, but you can split out a directory as a subtree at any given whim. If you decide you dont want something to be a subtree anymore... just stop performing subtree split or pushes.

The downside to subtrees is that they arent tracked at all. So you have to remember all the paths and their relationship to their repositories - and anyone else working on the project also just has to know that if they want to perform subtree operations. The good news, is most developers can just work on any code on any of the dependencies without worrying about how it will be pushed out to those repos. Also, as you said, some bash scripts can hel automate the manual stuff.

eddiemoya
  • 6,713
  • 1
  • 24
  • 34
  • I agree with your assessment of subtrees vs submodules, but it doesn't really address the question asked here. That said, the shortcomings you mentioned about subtrees, I think will be easily addressed with a contrib to make them write to the .git/config. I'm also sort of addressing some of the problems directly in TortoiseGit. – johnb003 Mar 12 '14 at 02:32
  • I think it does answer the question, or at least helping with it. What I'm saying is that with subtrees, it doesn't matter as much. You can make no subtrees at first, and turn your directories into subtrees later - at a whim. You can make a bunch now and just forget about them later if you dont need them. I'm saying subtrees let you worry less about making the right decision up front. – eddiemoya Mar 12 '14 at 02:35
  • And yes, I think the shortcomings of subtrees will be short lived. There are already scripts out there that solve the problem. – eddiemoya Mar 12 '14 at 02:37
  • Oh, in that case I completely agree, that's basically the same answer I gave :P +1 – johnb003 Mar 12 '14 at 02:38
1

When you have a good re-use case for multiple projects then consider splitting it out to a sub-project. I would avoid creating a shared project before you have two projects that use it.

Criteria I would use to consider making a sub-project repo:

  1. Is it used by multiple projects?
  2. Is it self contained?
  3. Does it change frequently?

I find subtrees the easiest to manage in that I can develop the library as part of a project and then split it off when the need arises.

I'd also just like to point out, it's perfectly okay for 2 projects to diverge on common libraries, and often preferred in order to keep them in a stable state. So long as it's easy to converge common code, I see no harm in taking a lazy approach to sharing libraries.

In any case, it's a good sign to have this problem; it means you have done a good job of making re-usable code. :)

johnb003
  • 1,821
  • 16
  • 30
1

When you're working in a distributed environment, giving the features of git, you should avoid to directly group different components into a single repository if those components are used by other projects or if you plan to do that. Or if it's either probable or desirable it will happen in the future.

This because developers/contributors will be able to focus on their part without the need to download the full history of every other components they're not going to use/change. Think at that is also crucial if you're working with contributors from countries/areas where internet speed is slower than the one we're used at.

As you tried and understand various methods you're not stuck with low knowledge and it shouldn't be hard a hard task. As far as I know you got all possible alternatives.

I won't worry about having dozens or potentially hundreds of smaller repositories if they're somehow independent from the main repository. Having so many repository will only increase the time of first configuration of your new main repository.

You should favor the big repository solution only if you need to migrate "immediately" from subversion. Or someone with no or low knowledge of alternatives.

I would use git subtree because it's available with git as standard features: users will not be required to install anything additional than git, and it will continue to stay around until git will.

David
  • 2,987
  • 1
  • 29
  • 35