6

I'm migrating some personal project repositories to Git from Mercurial. One of the projects relies on some non-changing, but large, shapefiles and SQLite databases These files are important and need to live inside the repo so that anyone checking out the project has access to them. With Mercurial, this was easy to deal with; I used the largefiles extension. largefiles automatically handled file additions/changes by not trying to analyze the content of files larger than X in size. That is, I could do hg addremove, and everything would just work.

Git, just like Mercurial, is not designed to track large files. However, I don't see a similar extension. I've looked into git-annex, but it seems like I need to manually keep track of the files (i.e., I can't just arbitrarily do git add -A). Also, if I'm reading this right, git-annex seems to maintain large files in a completely separate repo. I want to keep the large files in my current repo in the directories they currently live.

How do people handle this situation? Surely there are lots of projects that need to track large files integral to the operation of the project. Will git-annex accomplish this, or do I need some other extension?

Geoff
  • 2,461
  • 2
  • 24
  • 42
  • 1
    "largefiles automatically handled file additions/changes by not trying to analyze the content of files larger than X in size." -- how does this differ from `core.bigFileThreshold`? -- "Files larger than this size are stored deflated, without attempting delta compression. Storing large files without delta compression avoids excessive memory usage, at the slight expense of increased disk usage. Default is 512 MiB on all platforms. This should be reasonable for most projects as source code and other text files can still be delta compressed, but larger binary media files won’t be." – jthill Nov 22 '13 at 03:33
  • @jthill, thanks! I didn't know that was even an option! I'm shocked I haven't read about this before. I actually think that solves my problem exactly. If you toss this post into a separate answer, I'll mark it answered and give you the bounty. – Geoff Nov 22 '13 at 05:33

3 Answers3

5

The only one git-like system designed to deal with large (even very very large) files is:

bup (see more in GitMinutes #24)

The result is an actual git repo, that a regular Git command can read.

I detail how bup differs from Git in "git with large files".

Surely there are lots of projects that need to track large files integral to the operation of the project.

No there isn't. This is simply not what Git is designed for, and even git-annex is a workaround which isn't entirely satisfactory: see "git-annex with large files".
I mention other tools in "How to handle a large git repository?".

Community
  • 1
  • 1
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
2

largefiles automatically handled file additions/changes by not trying to analyze the content of files larger than X in size.

How does this differ from core.bigFileThreshold? --

core.bigFileThreshold

Files larger than this size are stored deflated, without attempting delta compression. Storing large files without delta compression avoids excessive memory usage, at the slight expense of increased disk usage.
Default is 512 MiB on all platforms. This should be reasonable for most projects as source code and other text files can still be delta compressed, but larger binary media files won’t be."

jthill
  • 55,082
  • 5
  • 77
  • 137
  • 2
    This is fine as long as the files never change and aren't deleted. If you do change files or delete them, then this won't prevent all clones from needing all versions of every large file - in other words, this really is very different from largefiles. In the OP's specific scenario (which strikes me as unusual), I don't really see the problem with just committing the files into git itself, really. – Eamon Nerbonne Dec 05 '13 at 23:29
  • @EamonNerbonne But this _is_ just committing the files into git itself, it's just telling git not to try deltifying them. If OP's "not trying to analyze the content of files larger than X" still leaves hg deltifying them, what analysis does its largefiles shut off? – jthill Dec 06 '13 at 21:15
  • largefiles only needs the current version of the files in the clones. It's like a transparent fallback to svn for large files, or in git terminology like a partially shallow clone. – Eamon Nerbonne Dec 08 '13 at 12:35
  • Yeah, this is perfect for my use case because the large files I need in my repo can be considered immutable - I generate them once and just read them after that. That's why I marked this as the answer. I'm actually surprised more people don't have this question. Certainly lots of other projects require relatively large input files. – Geoff Dec 09 '13 at 17:18
  • 1
    @Geoff: in my experience those large files aren't truly immutable; there will be new version or alternate datasets or whatever eventually, and then you want to avoid storing the old versions in every clone if possible. Shapefiles may need tweaking or just replacing, sqlite databases may need updating or upgrading... – Eamon Nerbonne Dec 11 '13 at 13:34
  • GIt 2.38 slightly amends `core.bigFileThreshold` description in [this commit](https://github.com/git/git/commit/3c3ca0b0c193e4f2cdf281b5e20ddb75c555f8e6). – VonC Jul 23 '22 at 20:18
0

I keep track of a md5 hash of large files as opposed to the files themselves. I also have a script that will go out and download the large files that is tracked in the repository.

I'm sure there are much nicer methods than this, but it works in a pinch.

RyPeck
  • 7,830
  • 3
  • 38
  • 58
  • The main problem with this method is that the SQLite databases are made by me, so they can't be found outside of the repo. – Geoff Nov 16 '13 at 23:42