6

Problem

Git hangs while writing objects for a 81 kB repo over HTTPS (SSH is not allowed). The specific error message from Nginx's access.log is as follow:

POST /Hello-World/.git/git-receive-pack HTTP/1.1" 504 183 "-" "git/2.28.0.windows.1"

Background

I host Nginx and Git servers that are inside a mounted Debian image. Important to note: my www for all my website files are mounted separately as read-only. Also my Git root folder is inside www (i.e., www/git).

sudo mount -o loop,offset=1 raspbian-stretch-lite.img /mnt/
sudo mount --bind www/ /mnt/usr/share/nginx/www/
sudo mount -o bind,remount,ro /mnt/usr/share/nginx/www

My Nginx conf is as follows:

server {
  listen 443 ssl;
  server_name git.domain.com;
  ssl_certificate fullchain.pem;
  ssl_certificate_key privkey.pem;

  # fast clone static files
  location ~ \.git/objects/(?:[0-9a-f]+/[0-9a-f]+|pack/pack-[0-9a-f]+\.(?:pack|idx))$ {
    root /usr/share/nginx/www/git;
    limit_except GET HEAD {
      allow all;
    }
  }

  location ~ ^.*\.git/(HEAD|info/refs|objects/info/.*|git-(upload|receive)-pack) {
    root /usr/share/nginx/www/git;
    auth_basic "Restricted";
    auth_basic_user_file /usr/share/nginx/www/auth;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
    fastcgi_param GIT_HTTP_EXPORT_ALL "";
    fastcgi_param GIT_PROJECT_ROOT $realpath_root;
    fastcgi_param REMOTE_USER $remote_user;
    fastcgi_param PATH_INFO $uri;
    fastcgi_param unix:/var/fcgiwrap.socket;
  }

  location /gitweb.cgi {
    root /usr/share/nginx/www/gitweb;
    include fastcgi_params:
    gzip off;
    fastcgi_param SCRIPT_NAME $uri:
    fastcgi_param GITWEB_CONFIG $realpath_root/gitweb.conf;
    fastcgi_pass unix:/var/fcgiwrap.socket;
  }

  location / {
    root /usr/share/nginx/www/gitweb;
    index gitweb.cgi;
  }
}

If I visit https://git.domain.com from browser, I get a Gitweb GUI. I clone with git clone https://git.domain.com/Hello-World/.git and all is well. But when I try to push using git push, it hangs at writing object step, eventually giving me a gateway timeout error. Interestingly, every time this happens, both https://git.domain.com Gitweb page and Git backend shut down, forcing me to kill and restart my fcgiwrap process.

My guess

I've tried git config --global http.postBuffer 524288000 to no avail. Since I'm authenticating (by typing my username and password), git-receive-pack should be allowed, right?

I'm assuming this has something to do with git folder not having write permission? Perhaps my Nginx config's regex is not correct?

Edits:

Normally errors from Git would go to stderr, which is usually written into a log by the web server. Is there such a log on your system, and if so, what does it contain?

The error I get in my command prompt from client-side is as follows:

(base) C:\Users\pairwiseseq\Hello-World>git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Writing objects: 100% (3/3), 258 bytes | 258.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly
Everything up-to-date

Also from Nginx's error.log:

*9 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 123.456.7.890, server: git.domain.com, request: "POST /Hello-World/.git/git-receive-pack HTTP/1.1", upstream: "fastcgi://unix:/var/fcgiwrap.socket", host: "git.domain.com"

strace for fcgiwrap process:

read(3, "\1\1\0\1\0\10\0\0\0\1\1\0\0\0\0\0\1\4\0\1\0041\7\0\f\0QUERY_STRING\16\4REQUEST_METHODPOST\f&CONTENT_TYPEapplication/x-git-receive-pack-request\16\3CONTENT_LENGTH452\v\"SCRIPT_NAME/Hello-World/.git/git-receive-pack\v\"REQUEST_URI/Hello-World/.git/git-receive-pack\f\"DOCUMENT_URI/Hello-World/.git/git-receive-pack\r\30DOCUMENT_ROOT/usr/share/nginx/www/git\17\10SERVER_PROTOCOLHTTP/1.1\16\5REQUEST_SCHEMEhttps\5\2HTTPSon\21\7GATEWAY_INTERFACECGI/1.1\17\fSERVER_SOFTWAREnginx/1.10.3\v\rREMOTE_ADDR123.456.7.890\v\5REMOTE_PORT39118\v\fSERVER_ADDR123.456.7.12\v\3SERVER_PORT443\v\20SERVER_NAMEgit.domain.com\17\3REDIRECT_STATUS200\17\"SCRIPT_FILENAME/usr/lib/git-core/git-http-backend\23\0GIT_HTTP_EXPORT_ALL\20\30GIT_PROJECT_ROOT/usr/share/nginx/www/git\v\10REMOTE_USERusername\t\"PATH_INFO/Hello-World/.git/git-receive-pack\24\4HTTP_ACCEPT_ENCODINGgzip\v\10HTTP_PRAGMAno-cache\17\31HTTP_USER_AGENTJGit/5.1.2.201810061102-r\22*HTTP_AUTHORIZATIONBasic cGc5eno0c2g6WWw5V1RpdnVrWlE2QTRNRw==\21&HTTP_CONTENT_TYPEapplication/x-git-receive-pack-request\v%HTTP_ACCEPTapplication/x-git-receive-pack-result\25\4HTTP_CONTENT_ENCODINGgzip\23\3HTTP_CONTENT_LENGTH452\t\20HTTP_HOSTgit.domain.com\17\nHTTP_CONNECTIONKeep-Alive\0\0\0\0\0\0\0\1\4\0\1\0\0\0\0\1\5\0\1\1\304\4\0\37\213\10\0\0\0\0\0\0\0\1\255\1R\37600b97fd1a60b01f91b314f59955a4e4d4e80d8edf11d f1a5d26dcb87f31b127cd397894aae43f86f362c refs/heads/master\0report-status delete-refs ofs-delta side-band-64k agent=JGit/5.1.2.201810061102-r0000PACK\0\0\0\2\0\0\0\3\234\fx\234\225\313\301\r\3020\f@\321{\246\310\2\225\354&v\32\t!\6\340\306\4I\354\10\16\220*r\367\247\214\300\345\35\276\364m\252zT&\26\222\36\326\232\2\244J\25\267\225\2651\"K<\273f\255\215\334^\246~\314\247.X\30*`\317X\3\306N9\23\225\250Q\242n \233JG\24W\16{\216\351m\234\\~\336\36\307\354\245\351r/\273\215}\tW\217\fk\f\t3\373\5\10\300\265\361~\277\314\364\337\317}\1q~<[\242\2x\234340031Q\10rut\361ue\210\233\301z.Q\341Ks\234\307\375\0O\311\233\5\355N\241:\0\257\246\v\211=x\234\363H\315\311\311WHLO\314\314S\344\2\0 d\4@\4{\334\"\301R\v\30\337\271\275\n\203\3349{D\255S;9\301\247\200\255\1\0\0\0\0\0\0\1\5\0\1\0\0\0\0", 8192) = 1584
write(5, "\37\213\10\0\0\0\0\0\0\0\1\255\1R\37600b97fd1a60b01f91b314f59955a4e4d4e80d8edf11d f1a5d26dcb87f31b127cd397894aae43f86f362c refs/heads/master\0report-status delete-refs ofs-delta side-band-64k agent=JGit/5.1.2.201810061102-r0000PACK\0\0\0\2\0\0\0\3\234\fx\234\225\313\301\r\3020\f@\321{\246\310\2\225\354&v\32\t!\6\340\306\4I\354\10\16\220*r\367\247\214\300\345\35\276\364m\252zT&\26\222\36\326\232\2\244J\25\267\225\2651\"K<\273f\255\215\334^\246~\314\247.X\30*`\317X\3\306N9\23\225\250Q\242n \233JG\24W\16{\216\351m\234\\~\336\36\307\354\245\351r/\273\215}\tW\217\fk\f\t3\373\5\10\300\265\361~\277\314\364\337\317}\1q~<[\242\2x\234340031Q\10rut\361ue\210\233\301z.Q\341Ks\234\307\375\0O\311\233\5\355N\241:\0\257\246\v\211=x\234\363H\315\311\311WHLO\314\314S\344\2\0 d\4@\4{\334\"\301R\v\30\337\271\275\n\203\3349{D\255S;9\301\247\200\255\1\0\0", 452) = 452
read(6, "Expires: Fri, 01 Jan 1980 00:00:00 GMT\r\nPragma: no-cache\r\nCache-Control: no-cache, max-age=0, must-revalidate\r\nContent-Type: application/x-git-receive-pack-result\r\n\r\n", 4096) = 165

After this, fcgiwrap just completely hangs. I can neither clone nor push at this point and have to spawn a new fcgiwrap process.

pairwiseseq
  • 313
  • 2
  • 13
  • Normally errors from Git would go to stderr, which is usually written into a log by the web server. Is there such a log on your system, and if so, what does it contain? Also, [setting `http.postbuffer`](https://git-scm.com/docs/gitfaq#Documentation/gitfaq.txt-WhatdoescodehttppostBuffercodereallydo) is ineffective and just wastes memory unless your HTTP server is broken, which nginx is not. – bk2204 Oct 09 '20 at 03:55
  • @bk2204, could you specify if this log is on server- or client-side, please? I've edited the post to include the `stdout` error message. There are also no error logs inside the repo if that's what you were referring to. – pairwiseseq Oct 09 '20 at 04:20
  • Any relevant logs in your server's `/var/log`? – jingx Oct 09 '20 at 04:41
  • @jingx, only the one in `/var/log/nginx/access.log`, which I've included at the top of my post. – pairwiseseq Oct 09 '20 at 04:48
  • Nothing in `/var/log/syslog` or `auth.log`? Also can you enable more detailed logging from nginx? The issue looks like the git process on the server broke down so in general you want to look into these server logs. – jingx Oct 09 '20 at 04:59
  • @jingx, nothing in `syslog` or `auth.log` unfortunately. Yes, I will add more logging info once I have it enabled in Nginx. – pairwiseseq Oct 09 '20 at 05:12
  • The log I'm referring to is server side. A directive like this might help to log the stuff you're expecting: `error_log /var/log/nginx/nginx_error.log warn;` – bk2204 Oct 09 '20 at 15:50
  • @bk2204, thank you for the suggestion. I've now added the warning to the updated post. – pairwiseseq Oct 11 '20 at 16:26
  • [this other answer](https://stackoverflow.com/a/50470075/86072) points to [this support page on bitbucket support](https://confluence.atlassian.com/bitbucketserverkb/git-push-fails-fatal-the-remote-end-hung-up-unexpectedly-779171796.html) : the "Workaround" section mentions the `client_max_body_size` nginx parameter. – LeGEC Oct 16 '20 at 07:26
  • 1
    To rule out permissions problem : if it is possible to open a temporary ssh access to that repo, you may try to push to this repo through ssh and see if it fails ; or, while logged to the server, `su` to the nginx user, and do a local clone + push. – LeGEC Oct 16 '20 at 07:36
  • @LeGEC, I've tried `client_max_body_size 0` solution but didn't work. Interestingly, I chrooted into my image as root. Cloned, added, and commited all fine. Again, when I tried to push, I got a time out error so it's definitely a permission problem. I had not thought about doing it locally, thank you. Like I mentioned above, the git root folder is mounted as read only. Maybe I need to allow chroot's root for write permission from outside my image? – pairwiseseq Oct 16 '20 at 18:08
  • Possibly. We're you able to try this ? – LeGEC Oct 17 '20 at 19:25

1 Answers1

0

Have you tried the answer from here?

git config --add remote.origin.proxy ""

Also there is an answer here with people complaining of 504 errors with a "push".

I also got HTTP 504 errors (gateway timeout) with push at first; for me, this was caused by a permission problem: fcgiwrap ran as user www-data, but the git repository directories were owned by another user.

Connor Dickson
  • 770
  • 3
  • 12