11

I'm having a problem where trying to do a git push hangs up while "Writing objects". This is not the common scenario of Windows and git-daemon that I have seen answered elsewhere. This is Linux to Linux, using ssh. On both the client and server, git version is 1.7.8.6.

I've tried doing the following:

  • Create a local empty repo and push to that -- works fine
  • Pull from local repo to local repo -- works fine
  • Clone the remote repo to the local machine -- works fine
  • git fsck and git gc on both server and client repos
  • Push to pre-existing remote repo -- hangs writing objects at 11%
  • Create empty remote repo and push to that -- hangs writing objects at 2%

There's an env variable I can set to make git produce debug messages. Nothing useful there.

I've talked to people at length in IRC about this, and I've tried googling it. We're getting nowhere. Can anyone here help me figure out what's going wrong?

Thanks!

UPDATE: Based on someone's suggestion, I did an strace -F, and it gets stuck on a select() in a child process. I'd provide the whole trace, but stackexchange doesn't have a way to upload. Anyhow, here's the relevant section:

[pid 14414] getsockopt(3, SOL_SOCKET, SO_RCVBUF, [87872], [4]) = 0^M
[pid 14414] select(7, [3 4], [3], NULL, NULL) = 2 (in [4], out [3])^M
[pid 14414] read(4,  <unfinished ...>^M
[pid 14433] <... write resumed> )       = 8192^M
[pid 14433] write(1, "\262\36\237\265\234\n\332W\26\271.\250\271W\34\250N\34*\341\247R\21\177\\Dq\23\27\211\270\25"..., 8192) = 8192^M
[pid 14433] write(1, "Y\343}z\367\31y\321)\254\236na\274\347*\253\247\264d\25\213\364\243\331\242\236\3222U3\246"..., 8192 <unfinished ...>^M
[pid 14414] <... read resumed> "t\277\177\276\361\377\327U\361Re\f\347\300\27\3\255}E\363\2277\312p\356F\335U\345\336\250\273"..., 16384) = 16384^M
[pid 14414] write(3, "\210y\r\202\303`\222m\216\177x \354\216Y\322n\232l\273\237\214\352\32\r\261r\311\347\356\236\26"..., 16432) = 16432^M
[pid 14414] getsockopt(3, SOL_SOCKET, SO_RCVBUF, [87872], [4]) = 0^M
[pid 14414] select(7, [3], [3], NULL, NULL) = 1 (out [3])^M
[pid 14414] write(3, "\v\267\355\33\257\332\330\326\233\260;\223\212z\333\213\21'1\20\7\366\"f\253\373\0059\24\206C4"..., 5504) = 5504^M
[pid 14414] getsockopt(3, SOL_SOCKET, SO_RCVBUF, [87872], [4]) = 0^M
[pid 14414] select(7, [3], [], NULL, NULL <unfinished ...>^M
[pid 14433] <... write resumed> )       = ? ERESTARTSYS (To be restarted if SA_RESTART is set)^M
[pid 14433] --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} ---^M
[pid 14433] rt_sigreturn()              = 1^M
[pid 14433] write(1, "Y\343}z\367\31y\321)\254\236na\274\347*\253\247\264d\25\213\364\243\331\242\236\3222U3\246"..., 8192) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)^M
[pid 14433] --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} ---^M
[pid 14433] rt_sigreturn()              = 1^M
[pid 14433] write(1, "Y\343}z\367\31y\321)\254\236na\274\347*\253\247\264d\25\213\364\243\331\242\236\3222U3\246"..., 8192) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)^M
[pid 14433] --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} ---^M
[pid 14433] rt_sigreturn()              = 1^M
[pid 14433] write(1, "Y\343}z\367\31y\321)\254\236na\274\347*\253\247\264d\25\213\364\243\331\242\236\3222U3\246"..., 8192) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)^M
[pid 14433] --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL, si_value={int=1, ptr=0x1}} ---^M
[pid 14433] rt_sigreturn()              = 1^M
Timothy Miller
  • 1,527
  • 4
  • 28
  • 48
  • This is an ssh problem. If I change the protocol to `git://`, then I can push remotely all I want. Same for `file://`. But using ssh, it hangs. Since I want ssh encryption, I still need to figure out why ssh is going wrong. – Timothy Miller May 26 '12 at 22:54
  • I tried sending a large file via sftp, and it stalls out at 160K. So this is the problem. SSH is broken. Unfortunately, googling for ssh stalls is getting me nowhere. – Timothy Miller May 28 '12 at 12:47

2 Answers2

7

The problem was a bug in ssh. A recent version broke something, and it stalls out during large transfers. There is a work-around.

The fix is to set "TcpRcvBufPoll no" in sshd_config.

http://forums.gentoo.org/viewtopic-t-925044-start-0-postdays-0-postorder-asc-highlight-.html

http://forums.gentoo.org/viewtopic-p-7036250.html

Timothy Miller
  • 1,527
  • 4
  • 28
  • 48
  • Couldn't find mention of [TcpRcvBufPoll](http://www.psc.edu/index.php/hpn-ssh/640) in very many places; it may not be available in all versions. +1 for heavy research into a poorly-documented solution. – Todd A. Jacobs Jun 02 '12 at 10:40
  • I can't take credit. I asked somewhere, and someone responded who had just happened to see the solution somewhere else. – Timothy Miller Jun 03 '12 at 12:17
  • Please tell me where to find sshd_config. I am using egit in eclipse – Imran Khurram Aug 25 '14 at 09:49
  • 1
    @ImranKhurram `find / 2> /dev/null | grep sshd_config` will find that file on your computer. On my computer it was in "/private/etc/sshd_config" – kris Oct 31 '15 at 00:23
0

Is there anything going on at the server? My first guess would be space problems.

After that, I'd check inodes (df -i).

talexb
  • 135
  • 1
  • 8