3

I have a Redmine server with Git on it. It has 2GB of RAM.

When someone tries to push a repository with data bigger than 2GB, I get the following error:

Counting objects: 957, done.
Delta compression using up to 12 threads.
Compressing objects: 100% (577/577), done.
Writing objects: 100% (957/957), 2.18 GiB | 29.66 MiB/s, done.
Total 957 (delta 255), reused 862 (delta 229)
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date

Before the first fatal: ..., ruby starts to take all the RAM on the server.

When reaching ~2GB of RAM usage, the fatal: ... errors occur. Error on the server:

App 9339 stderr: Errno::ENOMEM: Cannot allocate memory - git

So my guess is that Git or Redmine tries to put those 2.18 GB into RAM, which would obviously not work.

After the error:

  • the repository on the server is still empty;
  • the RAM usage is still at 95+% until I restart Redmine.

How can I make it work? Is it Redmine or Git that eats all the RAM?

Could pushing the data "part by part" (like here) be a solution?

I already tried to increase the HTTP post buffer size (git config http.postBuffer 524288000) without success.

EDIT (2017-06-01):

Some specs:

  • Ubuntu LTS 12.04.05
  • Redmine v2.6.9
  • Git v1.7.9.5

Redmine is deployed behind an Apache server (v2.2.22) with the passenger module. It uses Grack (last revision) for its Git integration with the Redmine SCM plugin to create repositories.

After re-looking at the logs, it looks like the culprit is Grack.

App 9339 stderr: Errno::ENOMEM: Cannot allocate memory - git
App 29461 stderr:       /opt/grack/lib/git_adapter.rb:24:in ``'
App 29461 stderr:       /opt/grack/lib/git_adapter.rb:24:in `command'
App 29461 stderr:       /opt/grack/lib/git_adapter.rb:34:in `upload_pack'
App 29461 stderr:       /opt/grack/lib/grack.rb:84:in `get_info_refs'
App 29461 stderr:       /opt/grack/lib/grack.rb:55:in `call'
App 29461 stderr:       /opt/grack/lib/grack.rb:55:in `call'

In other words, the error occurs when Grack launch a git upload-pack command.

EDIT (2017-06-01) (2):

The local repository had indeed a pack of 2+GB. I repacked it with the following:

git repack -a -d --window-memory=512m --max-pack-size=512m

To reduce that big pack into multiple, smaller packs. I also configured the following client and server side:

git config pack.windowMemory 512m
git config pack.packSizeLimit 512m

But the problem still occurs.

App 30005 stderr: [ 2017-06-01 09:15:04.4393 30122/0x000000019264a8(Worker 1) utils.rb:72 ]: *** Exception Errno::ENOMEM in Rack body object #each method (Cannot allocate memory - git) (process 30122, thread 0x000000019264a8(Worker 1)):
App 30005 stderr:       from /opt/grack/lib/git_adapter.rb:19:in `popen'
App 30005 stderr:       from /opt/grack/lib/git_adapter.rb:19:in `command'
App 30005 stderr:       from /opt/grack/lib/git_adapter.rb:43:in `receive_pack'
App 30005 stderr:       from /opt/grack/lib/grack.rb:70:in `block in service_rpc'

EDIT (2017-06-01) (3):

Added some logging on Grack to see what it exactly launches:

git receive-pack --advertise-refs --stateless-rpc /var/git/mygit.git

Since it does not launch the command from the Git repository directory, I tried the following:

git config --global pack.windowMemory 512m
git config --global pack.packSizeLimit 512m

Sadly still not working.

EDIT (2017-06-01) (4):

Tried again but with 64m instead of 512m, still no effects.

It looks like Git still needs to take all objects on RAM to start the packaging. Client side, the git repack command does indeed takes 2+GB to execute, even if the resulting output are multiple 512-MB files.

kagmole
  • 2,005
  • 2
  • 12
  • 27
  • Downvoters, please explain. I accept the fact that my question may be stupid but just `-1` will not help. – kagmole May 31 '17 at 13:16
  • I've up-voted you to correct your downvoter problem :D, please explain more, which OS, Which GIt/redmine version, how is it deployed, is it redminegit hosting plugin or what? – Aleksandar Pavić Jun 01 '17 at 06:13
  • Thank you for your concern. I updated my question. I will look for a potential problem with Grack now. – kagmole Jun 01 '17 at 06:37
  • Well I recommend you migrate to newer Redmine and use redminegithosting from jboxweb plugin... – Aleksandar Pavić Jun 01 '17 at 10:52
  • 1
    This has nothing to do with Redmine (which you use as an authenticator only) but only with how git handles and stores objects. In general, you should not store that large files in Git since it results in exactly these issues. Instead, you should store the large files outside of git or use something like [Git LFS](https://git-lfs.github.com/). For the latter though, there is no support I know of in Redmine or one of the common plugins right now. – Holger Just Jun 02 '17 at 14:20
  • @HolgerJust Yes Git LFS would be great sadly the repository was not created on our server first, we are a new remote, and the developers did not use Git LFS. The `git receive-pack` is the command taking all RAM and it looks like there is no way to tell it to not eat all the RAM. – kagmole Jun 02 '17 at 14:27

1 Answers1

0

Since this question is descending into the depths, here is the workaround I used as an answer.

It looks like receive/upload-pack needs to load everything on the RAM. Maybe I am wrong, but I found nothing to reduce the RAM usage into small packets. So what I did is creating a transient swap file.

# create a file of 4GB
sudo dd if=/dev/zero of=/swap bs=1024 count=4194303

# make it a swap file
sudo mkswap /swap

# enable the swap on this swap file
sudo swapon /swap

2GB was not enough: the push took like ~3.5GB, thus the 4GB swap file.

The push took a long time (no sh*t Sherlock) but at least it succeeded.

I left the new swap file for future similar problems, but if you want to delete it after the push you can do the following:

# disable the swap (will automatically move used memory into RAM or another swap)
sudo swapoff /swap

# delete the file
sudo rm /swap

Reference on UNIX guide

kagmole
  • 2,005
  • 2
  • 12
  • 27