5

My company is considering implementing Git but I have a question about what the best way would be to set it up. We have 3 sites and are planning on using Gerrit2 to create mirrors. Our repository is about 2GB and we would like to start adding binaries to it. I'm concerned about the space usage though. I don't mind if all versions of the binaries are stored in a handful of locations but I want to make sure that they don't bog down clone operations.

I understand that Git uses hard links but I think that will only work if we place a copy of the repository on every mount. Are there better options and if so what are the tradeoffs? Options that I'm looking at are "--shared" and "--reference".

dromodel
  • 9,581
  • 12
  • 47
  • 65

4 Answers4

6

Another alternative to git media mentioned by Marcelo is git annex:

See what git-annex is not:

git-annex is not git-media, although they both approach the same problem from a similar direction. I only learned of git-media after writing git-annex, but I probably would have still written git-annex instead of using it.

Currently, git-media has the advantage of using git smudge filters rather than git-annex's pile of symlinks, and it may be a tighter fit for certain situations.
It lacks git-annex's support for widely distributed storage, using only a single backend data store.
It also does not support partial checkouts of file contents, like git-annex does.

Note: abdelsaid adds in the comments:

You can use git-annex with bup (bup allows you to have versions), see git-annex/ special remotes/ bup (and Using bup)

I have presented bup in more details in "git with large files"

Community
  • 1
  • 1
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • Doesn't seem to allow you to keep multiple revisions of binaries, only store those that you have separately. – dromodel May 11 '11 at 02:02
  • I know this is old comment but if somebody is coming here, you can use git-annex with bup (bup allows you to have versions), see https://git-annex.branchable.com/special_remotes/bup/ – vimdude Apr 27 '14 at 15:40
  • @abdelsaid good update. I have included your comment in the answer for more visibility, and i have added a link to another answer of mine where I talked about bup. – VonC Apr 27 '14 at 15:44
3

Use git-media for large binaries. It stores references to binaries as SHA1 sums and hosts the binaries themselves in a place (and protocol) of your choosing. When you do a clone, it only fetches the binaries necessary to checkout the working copy.

Marcelo Cantos
  • 181,030
  • 38
  • 327
  • 365
  • That could complicate syncing to all sites as Gerrit2 wouldn't pick them up. Partial checkout support also seems to be missing, per comments below. Interesting option though. – dromodel May 11 '11 at 01:53
3

To just use native git, use a separate repo to house the binaries via git submodules. This has worked for me on IVR systems which had a ton of gigantic .wav files. If you need further clarification, feel free to contact me.

Here's a good write up on them:

http://progit.org/book/ch6-6.html

hope this helps

Adam Dymitruk
  • 124,556
  • 26
  • 146
  • 141
  • Submodules look interesting but how do they interact with partial checkouts? Can I specify that I want to check out only a portion of the submodule or do I have to check out the entire thing? – dromodel May 11 '11 at 01:48
  • that can be handled with special branches within the submodule repo that would have the files you need. – Adam Dymitruk May 11 '11 at 07:15
2

Yet another possibility is git-fat (or Cyan's fork of git-fat with some enhancements); it is even lighter weight (depends only on Python and rsync) than git-media (Ruby), and definitely lighter than git-annex (Haskell). The cost of that is less configurability, but this may be the best solution in some cases.

Alex Dupuy
  • 5,984
  • 3
  • 39
  • 35