80

Have run into a problem with repository and tried almost every possible config setting found out there eg. pack.WindowMemory etc etc

I believe someone has checked in a large file to remote repository and now each time I try and pull or push to it, GIT tries to pack it and runs out of memory:

Auto packing the repository for optimum performance. You may also
run "git gc" manually. See "git help gc" for more information.
Counting objects: 6279, done.
Compressing objects: 100% (6147/6147), done.
fatal: Out of memory, malloc failed (tried to allocate 1549040327 bytes)
error: failed to run repack

Have tried git gc & git repack with various options but keeps returning same error.

Almost given up and about to just create a new repo but thought I'd ask around first :)

timothyclifford
  • 6,799
  • 7
  • 57
  • 85
  • [Maybe related](http://stackoverflow.com/questions/4826639/repack-of-git-repository-fails). Have you managed to solve yours, tho? – cregox Aug 17 '12 at 17:51
  • In our case, we solved it with my link on a 7 gb repo. My suggestion, if you can't go with GIT, is trying Hg or Perforce. – cregox Aug 19 '12 at 15:40
  • I think it's a bug in MsGit rather than an issue with Git itself. Don't think it warrants moving away from Git altogether either, especially not to Mercurial! – timothyclifford Aug 20 '12 at 02:08
  • if so, can't u get an ubuntu pen drive just to try it? – cregox Aug 20 '12 at 13:35
  • an update, I've made a new repository with new files, which has 3gb and nothing from the link solved in my case here... Only thing that did help was `git config --global pack.threads 1` cited below. That worked. But anyway... Have you tried the ubuntu pen drive? – cregox Aug 29 '12 at 21:38
  • possible duplicate of [Git gc using excessive memory, unable to complete](http://stackoverflow.com/questions/8214321/git-gc-using-excessive-memory-unable-to-complete) – crashmstr Feb 21 '14 at 17:54

6 Answers6

116

I found a solution Here that worked for me.

In .git/config file (client and/or server) I added the following:

[core]
  packedGitLimit = 128m
  packedGitWindowSize = 128m

[pack]
  deltaCacheSize = 128m
  packSizeLimit = 128m
  windowMemory = 128m
git
  • 1,184
  • 1
  • 8
  • 2
  • 2
    It looks like msysgit does not use the correct default values for 32 bit if run on an 64 bit machine. However it should, as the client is 32 bit. These values worked for me: [core] packedGitLimit = 256m packedGitWindowSize = 32m [pack] deltaCacheSize = 256m packSizeLimit = 1g windowMemory = 256m – fmuecke May 26 '14 at 09:44
  • 1
    I had similar problem and in my case disabling the delta compression for zips and a custom binary compressed file worked. In .git/info/attributes, I've added "*.zip binary -delta". – MartinTeeVarga Jul 06 '15 at 23:19
  • 128m seems a bit low. is that per-thread? if so it does make a lot more sense, as if 32-bit git were to allocate more than 4096m across all threads, of course it would crash – Force Gaia Jul 10 '18 at 14:10
  • Helped on Linux too, for git lfs, when average file size is ~100 mb, commit size is ~8gb. – Vitaly Zdanevich Nov 24 '19 at 07:57
21

For reference (you might already seen it), the msysgit case dealing with that issue is the ticket 292.

It suggests several workarounds:

To disable the delta compression for certain files, in .git/info/attributes, add:

*.zip binary -delta

From Gitattributes man page:

Delta compression will not be attempted for blobs for paths with the attribute delta set to false.

Maybe a simpler workaround would be to somehow reset the history before that large file commit, and redo the other commits from there.

Community
  • 1
  • 1
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
15

EDIT:  Since git-v2.5.0 (Aug/2015), git-for-windows (formerly MSysGit)
      provides 64-bits versions as noticed by Pan.student.
      In this answer I was advising to install Cygwin 64-bits (providing 64-bits Git version).

I got a similar Out of memory, malloc failed issue using MSysGit when reaching the 4GB barrier:

> git --version
git version 1.8.3.msysgit.0

> file path/Git/cmd/git
path/Git/cmd/git: PE32 executable for MS Windows (console) Intel 80386 32-bit

> time git clone --bare -v ssh://linuxhost/path/repo.git
Cloning into bare repository 'repo.git'...
remote: Counting objects: 1664490, done.
remote: Compressing objects: 100% (384843/384843), done.
remote: Total 1664490 (delta 1029586), reused 1664490 (delta 1029586)
Receiving objects: 100% (1664490/1664490), 550.96 MiB | 1.55 MiB/s, done.
Resolving deltas: 100% (1029586/1029586), done.
fatal: Out of memory, malloc failed (tried to allocate 4691583 bytes)
fatal: remote did not send all necessary objects

real    13m8.901s
user    0m0.000s
sys     0m0.015s

MSysGit crashing after reaching 4 GB barrier

Finally git 64 bits from Cygwin fix it:

> git --version
git version 1.7.9

> file /usr/bin/git
/usr/bin/git: PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows

> time git clone --bare -v ssh://linuxhost/path/repo.git
Cloning into bare repository 'repo.git'...
remote: Counting objects: 1664490, done.
remote: Compressing objects: 100% (384843/384843), done.
remote: Total 1664490 (delta 1029586), reused 1664490 (delta 1029586)
Receiving objects: 100% (1664490/1664490), 550.96 MiB | 9.19 MiB/s, done.
Resolving deltas: 100% (1029586/1029586), done.

real    13m9.451s
user    3m2.488s
sys     3m53.234s

git 64 bits from Cygwin succeeded

FYI on linuxhost 64 bits:

repo.git> git config -l
user.email=name@company.com
core.repositoryformatversion=0
core.filemode=true
core.bare=true

repo.git> git --version
git version 1.8.3.4

repo.git> uname -a
Linux linuxhost 2.6.32-279.19.1.el6.x86_64 #1 SMP Sat Nov 24 14:35:28 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

If my answer does not fix your issue, you may also check these pages:

Community
  • 1
  • 1
oHo
  • 51,447
  • 27
  • 165
  • 200
  • I have used 64bit git from git-scm.com to solve out of memory error when checking out branch on large repositary – rluks Oct 28 '15 at 22:03
  • 1
    installing the 64bit version solved the “Out of memory - malloc failed” error when I was doing a rebase in a repository with "large" "*.lib" files (20-100mb+ each one). – Robson Dec 17 '15 at 10:51
1

Some of the options suggested in the selected answer seem to be only partially relevant to the issue or not necessary at all.

From looking at https://git-scm.com/docs/git-config, it appears that just setting the following option is sufficient (set only for the project here):

git config pack.windowMemory 512m

From the manual:

pack.windowMemory

The maximum size of memory that is consumed by each thread in git-pack-objects[1] for pack window memory when no limit is given on the command line. The value can be suffixed with "k", "m", or "g". When left unconfigured (or set explicitly to 0), there will be no limit.

With this, I never went over the specified 512m per thread, actually used RAM was about half of that most of the time. Of course, the amount chosen here is user-specific depending on the available RAM and number of threads.

Adrian
  • 111
  • 1
  • 4
0

In my case the config changes above didn't help. What helped was a simple server reboot (in my case sudo shutdown -r now). Looks like something was eating lots of memory on the server. Hope this helps someone, too.

Denis P
  • 497
  • 6
  • 11
-1

This worked for me, but I had to set the options via the command line using:

git --global core\pack [param] value
Pang
  • 9,564
  • 146
  • 81
  • 122
craig Rickett
  • 428
  • 1
  • 5
  • 17