2

We have a single git repo with 4 c++ projects which share a lot of source code files. These are our git branches:

  • master: used for the main development (currently working on proj2 version 4.x)
  • backports/proj1_4.x: branched-out from master when we released the proj1 version 4.0, it contains bugfixes and was used for releasing versions 4.1, 4.2
  • backports/proj2_3.x: same for proj2, version 3.x
  • backports/proj3_2.x: ...
  • backports/proj4_2.x: ...

When we fix a bug in master, we cherry-pick the fix to all backport branches. Having a single clone of the repository means that every time we switch between those branches, lots of c++ object files need to be recompiled.

Existing solutions

  • multiple full clones
    • checkout a different branch in each clone.
    • requires a lot of disk space (our .git folder has ~700MB), but avoids switching between branches and therefore also the recompilation of object files. This may be the simplest solution.
  • single clone with pre and post checkout git hooks for storing and restoring object files or timestamps of source files: https://stackoverflow.com/a/3195312/1341914
  • git worktree
  • ccache
    • when working on master for a long time, the cache dir may contain only object files from master, because the older object files from backport branches didn't fit in the ccache storage limit.
  • full clone of master + shallow clones of backport branches
    • how would we cherry-pick a commit when the clone has just depth 1? The commit SHA1 wouldn't be found in the shallow clone, but we could export the commit as a patch file and then apply it to the shallow clone
  • shared / reference
    • git clone --reference DIR GIT_REPO_URL or git clone --shared DIR
    • because the parent repo doesn't know about its other clones, when an object is removed in the main repo, it may become unreferenced in the child repo, potentially corrupting the child repo.

What would you recommend? Is there any other/better solution?

Michal Fapso
  • 1,242
  • 1
  • 15
  • 21
  • Hint: a shallow clone can be deeper than 1. The default depth at commercial CI/CD services (Travis at al) is 50, e.g. – phd Jun 24 '20 at 13:41
  • @phd, you're right, but in our case, the master branch is often hundreds of commits ahead of backport branches, so we still can't simply cherry-pick a master commit to a backport branch. – Michal Fapso Jun 24 '20 at 13:53
  • A full clone of `master` with a lot of shallow clones for backports? – phd Jun 24 '20 at 13:55
  • @phd, yes, exactly. Thanks, I've clarified it in the question now. – Michal Fapso Jun 24 '20 at 13:57

1 Answers1

1

We successfully use ccache in several projects.

I would suggest that you try with ccache (very easy to install and use). Make first one compilation and then check the cache size used

ccache -s

After that set the maximum cache size to double that of what was used in the first compilation and then try out your cherry-pick behaviour and check the ccache stats again. Do needed tuning.

This might in the end not be the best solution for you, but ccache is so easy to install and use (and doesnt need any changed behaviour from normal developers) so it is a low hanging fruit.

Eben
  • 157
  • 4
  • Thanks for your answer, @Eben. I'll give ccache a try. It could make sense even in combination with another approach. – Michal Fapso Jul 01 '20 at 10:41
  • I have to admit, @Eben, that using ccache has by far the best effort/outcome ratio. I just increased the ccache space from 5GB to 10GB and enabled compression. It seems that for our project, this keeps those backport branches in the ccache's cache for a long enough time, so that they really don't need to be recompiled when switching to them. – Michal Fapso Nov 16 '20 at 07:36