32

I have heard some claims that SVN handles binary files better than Git/Mercurial. Is this true and if so then why? As far as I can imagine, no version control system (VCS) can diff and merge changes between two revisions of the same binary resources.

So, aren't all VCS's bad at handling binary files? I am not very aware of the technical details behind particular VCS implementations so maybe they have some pros and cons.

Peter O.
  • 32,158
  • 14
  • 82
  • 96
Tower
  • 98,741
  • 129
  • 357
  • 507
  • 22
    I suggest that this *not* be closed as "not constructive" as a couple have suggested. There *are* solid facts out there about the different ways in which binary files are stored, and the way checkouts in DVCS store everything, et cetera. There will be some debate about what is better, but I think there is value in a question like this. – Chris Morgan Jul 06 '11 at 15:44
  • There are [a lot](http://stackoverflow.com/questions/3601672/git-and-binary-file) [of](http://stackoverflow.com/questions/2983635/can-git-mercurial-svn-or-other-version-control-tools-work-well-when-project-tr) [very close](http://stackoverflow.com/questions/1126239/git-or-subversion-for-binary-files) [questions](http://stackoverflow.com/questions/3037664/does-any-version-control-system-like-svn-git-or-mercurial-let-you-keep-latest) [around](http://stackoverflow.com/questions/5234318/version-control-for-large-binary-files-and-1tb-repositories) ... – Francois G Jul 06 '11 at 15:46
  • ... and I [do mean](http://stackoverflow.com/questions/275147/svn-optimizations-to-keep-large-repositories-responsive) [a whole](http://stackoverflow.com/questions/540535/managing-large-binary-files-with-git) [lot](http://stackoverflow.com/questions/5025998/versioning-when-binary-and-text-files-are-involved). – Francois G Jul 06 '11 at 15:49
  • 4
    @huitseeker: but it still wouldn't be quite fair to close it as an exact duplicate. Some of those are similar, but I wouldn't consider any of them to be really the same. – Chris Morgan Jul 06 '11 at 15:51

5 Answers5

21

The main pain point is in the "Distributed" aspect of any DVCS: you are cloning everything (the all history of all files)

Since binaries aren't stored in delta for most of them, and aren't compressed as well as text file, if you are storing rapidly evolving binaries, you end up quickly with a large repository which becomes much cumbersome to move around (push/pull).

For Git for instance, see What are the git limits?.

Binaries aren't a good fit for the feature a VCS can bring (diff, branch, merge), and are better managed in an artifact repository (like a Nexus for example).
This is not necessary the case for a CVCS (Centralized VCS) where the repository could play that role and be a storage for binaries (even if its not its primary role)

Community
  • 1
  • 1
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • 2
    Very well stated. However, I don't like putting binaries in any VCS. This isn't to say that Git and SVN don't handle binaries very well, but as you stated: They'll take up a lot of room. In Git especially since you download the entire repository on a _checkout_. Also, this isn't an all or nothing rule. Binary files like GIFs and JPEGs probably should be stored in the version control system. What you don't want to store are build results. – David W. Jul 06 '11 at 17:35
  • You don't *have* to clone everything though. I know that Mercurial lets you specify how far back in history you want to clone ("Clone to revision" in TortoiseHG). This is a great way to keep the size of your local repositories down. This feature alone is generally good enough for providing a consistent experience with repo's that contain a moderate binary file turnover. – nbevans Jul 06 '11 at 21:51
  • 2
    @NathanE I think you're misunderstanding Clone to Revision -- it's from 0 to the revision you specify. So it's true you could grab only the oldest few versions, but if you want what's current you need to grab everything. – Ry4an Brase Jul 07 '11 at 02:11
14

One clarification about git and binary files.

Git is compressing binary files as well as text files. So git is not crap at handling binary files as someone suggested.

Any file that Git adds will be compressed into loose objects. It doesn't matter if they are binary or text. If you have a binary or text file and you commit it, the repository will grow. If you make a minor change to the file and commit again your repository will grow again at approximately the same amount depending on the compression ratio.

Then you make a git gc. Git will find similarities in the binary or text files and compress them together. You will have a good compression if the similarities are large. If, on the other hand there are no similarities between the files, you will not have much of a gain compressing them together compared to compressing them individually.

Here is a test with a bit-mapped picture (binary) that I changed a little: 

martin@martin-laptop:~/testing123$ git init  
Initialized empty Git repository in /home/martin/testing123/.git/  
martin@martin-laptop:~/testing123$ ls -l   
total 1252  
-rw------- 1 martin martin 1279322 Jan  8 22:42 pic.bmp  
martin@martin-laptop:~/testing123$ git add .  
martin@martin-laptop:~/testing123$ git commit -a -m first  
[master (root-commit) 53886cf] first  
 1 files changed, 0 insertions(+), 0 deletions(-)  
 create mode 100644 pic.bmp  

// here is the size:  
martin@martin-laptop:~/testing123$ du -s .git  
1244    .git  

// Changed a few pixels in the picture  

martin@martin-laptop:~/testing123$ git add .  
martin@martin-laptop:~/testing123$ git commit -a -m second  
[master da025e1] second  
 1 files changed, 0 insertions(+), 0 deletions(-)  

// here is the size:  
martin@martin-laptop:~/testing123$ du -s .git  
2364    .git  

// As you can see the repo is twice as large  
// Now we run git gc to compress  

martin@martin-laptop:~/testing123$ git gc  
Counting objects: 6, done.  
Delta compression using up to 2 threads.  
Compressing objects: 100% (4/4), done.  
Writing objects: 100% (6/6), done.  
Total 6 (delta 1), reused 0 (delta 0)  

// here is the size after compression:  
martin@martin-laptop:~/testing123$ du -s .git  
1236    .git  

// we are back to a smaller size than ever...
Peter O.
  • 32,158
  • 14
  • 82
  • 96
martin
  • 169
  • 1
  • 2
9

Git and Mercurial both handle binary files with aplomb. Thet don't corrupt them, and you can check them in and out. The problem is one of size.

Source usually takes up less room than binary files. You might have 100K of source files that build a 100Mb binary. Thus, storing a single build in my repository could cause it to grow 30 times its size.

And it's even worse:

Version control systems usually store files via some form of diff format. Let's say I have a file of 100 lines and each line averages about 40 characters. That entire file is 4K in size. If I change a line in that file, and save that change, I'm only adding about 60 bytes to the size of my repository.

Now, let's say I compiled and added that 100Mb file. I make a change in my source (maybe 10K or so in changes), recompile, and store the new binary build. Well, binaries don't usually diff very well, so it's very likely I'm adding another 100Mb of size to my repository. Do a few builds, and my repository size grows to several gigabytes in size, yet the source portion of my repository is till only a few dozen kilobytes.

The problem with Git and Mercurial is that you normally checkout the entire repository onto your system. Instead of merely downloading a few dozen kilobytes that can be transfered in a few seconds, I am now downloading several gigabytes of builds along with the few dozen kilobytes of data.

Maybe people say Subversion is better since I can simply checkout the version I want in Subversion and not download the whole repository. However, Subversion doesn't give you an easy way to remove obsolete binaries from your repository, so your repository will grow and grow anyway. I still don't recommend it. Heck, I don't even recommend it even if the revision control system does allow you to remove old revisions of obsolete binaries. (Perforce, ClearCase, and CVS all do). It's just ends up being a big maintenance headache.

Now, this isn't to say you shouldn't store any binary files. For example, if I am making a web page, I probably have some gifs and jpegs that I need. No problem storing those in either Subversion or Git/Mercurial. They're relatively small, and probably change a lot less than my code itself.

What you shouldn't store are built objects. These should be stored in a release repository and fetched as needed. Maven and Ant w/ Ivy does a great job of this. And, you can use the Maven repository structure in C, C++, and C# projects too.

Martin Geisler
  • 72,968
  • 25
  • 171
  • 229
David W.
  • 105,218
  • 39
  • 216
  • 337
2

In Subversion you can lock binary files to make sure that nobody else can edit them. This mostly assures you that nobody else will modify that binary file while you have it locked. Distributed VCSs don't (and can't) have locks--there's no central repository for them to be registered at.

robert
  • 33,242
  • 8
  • 53
  • 74
  • The importance of this cannot be overstated. In most workflows, when someone is working on a binary file, they lock it. This actively prevents other people from doing so. Without that prevention, what will happen is that one person will check it in, and the other person will have to redo their work. That is not acceptable. – Nicol Bolas Jul 07 '11 at 00:58
  • 1
    While locking is useful to some extent, it does not prevent people from opening the local binary files and modifying them. – Tower Jun 10 '12 at 15:27
  • @rfactor if you set the `svn:needs-lock` property then Subversion makes the file read-only until you acquire the lock. – robert Jun 10 '12 at 15:40
0

Text files have a natural line-oriented struture that binary files lack. This is why it's harder to compare them using common text tools (diff). While it should be possible, the advantage of readability (the reason we use text as our preferred format in the first place) would be lost when applying diffs to binary files.

As to your suggestion that all version control systems "are crap at handling binary files", I don't know. In principle, there's no reason why a binary file should be slower to process. I would rather say that the advantages of using a VCS (tracking, diffs, overview) are more apparent when handling text files.

jforberg
  • 6,537
  • 3
  • 29
  • 47
  • While binary files might be difficult to manage by classic text focussed `diff`, some tools like TortoiseSVN provide scripts to forward some well known binary files to their `diff`-capable apps. That happens e.g. for ODT- and DOC-files, because LibreOffice Writer and MS Word are both capable to show changes between different versions of their files. Some image formats are compared by TortoiseSVN itself simply by showing both versions of the image side-by-side. So handling at least some binary files is pretty easy these days in some SCM-clients. – Thorsten Schöning Sep 19 '19 at 12:20