8

Using git-lfs/1.1.0 (GitHub; linux 386; go 1.5.1), file size 4.3G.

git init
git lfs install
git lfs track *.nnex

.gitattributes: *.nnex filter=lfs diff=lfs merge=lfs -text

git add evernote-backup.nnex: fatal: Cannot handle files this big

git lfs ls-files: Git can't resolve ref: "HEAD"

git lfs track: Listing tracked paths evernote-backup.nnex .gitattributes)

git lfs env:

WARNING: Reading LFS config from ".gitconfig", not ".lfsconfig". Rename to ".lfsconfig" before Git LFS v2.0 to remove this warning.
git-lfs/1.1.0 (GitHub; linux 386; go 1.5.1)
git version 2.1.4

LocalWorkingDir=/home/vitaly
LocalGitDir=/home/vitaly/.git
LocalGitStorageDir=/home/vitaly/.git
LocalMediaDir=/home/vitaly/.git/lfs/objects
TempDir=/home/vitaly/.git/lfs/tmp
ConcurrentTransfers=3
BatchTransfer=true
git config filter.lfs.smudge = "git-lfs smudge %f"
git config filter.lfs.clean = "git-lfs clean %f"

I am getting the following error:

git-lfs: fatal: Cannot handle files this big (4.3G)
Will
  • 24,082
  • 14
  • 97
  • 108
Vitaly Zdanevich
  • 13,032
  • 8
  • 47
  • 81
  • Isn't the inability for git to find HEAD a concern? http://stackoverflow.com/questions/4848607/git-fatal-no-such-ref-head – rbatt Dec 13 '15 at 02:11
  • @rbatt from your link I tried `echo ref: refs/heads/master >.git/HEAD` but after `git lfs ls-files` I see the same `Git can't resolve ref: "HEAD"` – Vitaly Zdanevich Dec 13 '15 at 04:02
  • can you do other things in git? Than unresolved HEAD seems like the problem; but I don't know. – rbatt Dec 13 '15 at 04:18
  • @rbatt it looks like I CAN to do other things in git, for example I created little test file, add it to git and commit – Vitaly Zdanevich Dec 13 '15 at 13:34
  • OK. I've used Git LFS a lot, but for Git in general, I'm not the most experienced user. I wonder if this question could be rephrased to focus on that HEAD issue ... I bet many more people will be able to answer. As for me, I'm unsure. – rbatt Dec 13 '15 at 16:45
  • Until you do a `git commit` `git lfs ls-files` will return `Git can't resolve ref: "HEAD"` because there is no `HEAD` yet. That being said: You're using a nearly 3 year old version of git, is there a reason you haven't upgraded? – Guildencrantz Dec 25 '15 at 05:43
  • @Guildencrantz I am at Debian 8.2 i386 and I installed `git` using `sudo apt-get install`. You think that I need to `sudo apt-get purge git` and install latest version from official site? – Vitaly Zdanevich Dec 25 '15 at 16:30
  • 2
    It might be an issue with used syscalls and size of data types used to represent sizes and offsets. If the threshold is at 4 GB, 32b infrastructure is probably at fault. You can use (pseudo)random binary data (almost incompressible) and bisection to find the maximum size that will go through. – Palec Dec 27 '15 at 11:29
  • Related on [SO]: [How to fix “fatal: Cannot handle files this big” in git?](http://stackoverflow.com/q/31388667/2157640), [git with large files](http://stackoverflow.com/q/17888604/2157640), [What are the file limits in Git (number and size)?](http://stackoverflow.com/q/984707/2157640) – Palec Dec 28 '15 at 08:11
  • Interesting content I found elsewhere: [pack-files > 2 GiB not working @ libgit2 issues](https://github.com/libgit2/libgit2/issues/534) (mentions trouble with `off_t` and was the inspiration for my first comment), [Working with large files @ GitHub Help](https://help.github.com/articles/working-with-large-files/) (useful for GitHub users, gives general advice), [Git Fails On Large Files @ Artem Dinaburg's Blog](http://blog.dinaburg.org/2013/07/git-fails-on-large-files.html) (related issues and overall interesting read). – Palec Dec 28 '15 at 08:17

1 Answers1

4

This is a 32bit addressing issue on i386, and Git and git-lfs simply cannot address a file larger than 4 GB. The maximum value of a 32-bit unsigned integer is 4,294,967,295, which comes out to about 4 GB.

We can see where this error is thrown inside the Git source code in git-compat-util.h:

744 static inline size_t xsize_t(off_t len)
745 {
746     if (len > (size_t) len)
747         die("Cannot handle files this big");
748     return (size_t)len;
749 }

I don't know enough about how git-lfs works internally to know if this can be worked around.

If working on a 64bit (x86_64) system rather than the 32bit (i386) system you're using is an option, that will fix your problem. Alternatively, you might be able to use git-annex instead of git-lfs with some success, but someone else had a similar issue with git-annex. There are not enough details in the bug report to know if this is still an issue on 32bit systems.

Unfortunately, you're facing a common limitation of 32bit hardware, and you'll run into a number of issues trying to handle files larger than 4 GB on these systems. It's upgrade time!

Palec
  • 12,743
  • 8
  • 69
  • 138
Will
  • 24,082
  • 14
  • 97
  • 108
  • this is still an issue with git-annex today, but most 32-bit programs can access 64-bit files nowadays, using that off_t structure. there is likely a way to fix this that isn't too hard. – fuzzyTew Feb 25 '21 at 12:28