Git sh.exe process forking issue on windows XP, slow?

Question

Git is essential to my workflow. I run MSYS Git on Windows XP on my quad core machine with 3GB of RAM, and normally it is responsive and zippy.

Suddenly an issue has cropped up whereby it takes >30 seconds to run any command from the Git Bash command prompt, including ls or cd. Interestingly, from the bash prompt it looks likes ls runs fairly quickly, I can then see the output from ls, but it then takes ~30 seconds for the prompt to return. If I switch to the windows command prompt (by running cmd from the start menu) git related commands also take forever, even just to run. For example git status can take close to a minute before anything happens. Sometimes the processes simply don't finish.

Note that I have "MSYS Git" installed as well as regular "MSYS" for things like MinGW and make.

I believe the problem is related to sh.exe located in C:\Program Files\Git\bin. When I run ls from the bash prompt, or when I invoke git from the windows prompt, task manager shows up to four instances of sh.exe processes that come and go.

Here I am waiting for ls to return and you can see the task manager has git.exe running and four instances of sh.exe:

If I ctrl-c in the middle of an ls I sometimes get errors that include:

sh.exe": fork: Resource temporarily unavailable
      0 [main] sh.exe" 1624 proc_subproc: Couldn't duplicate my handle<0x6FC> fo
r pid 6052, Win32 error 5
sh.exe": fork: Resource temporarily unavailable

Or for git status: $ git status

sh.exe": fork: Resource temporarily unavailable
sh.exe": fork: Resource temporarily unavailable
sh.exe": fork: Resource temporarily unavailable
sh.exe": fork: Resource temporarily unavailable

Can I fix this so that git runs quickly again, and if so how?

Things I have tried:

• Reboot
• Upgrade MSYS Git to most recent version & Reboot
• Upgrade MSYS to most recent version & Reboot
• Uninstall MSYS & uninstall and reinstall MSYS Git alone & Reboot

I'd very much like to not wipe my box and reinstall Windows, but I will if I can't get this fixed. I can no longer code if it takes me >30 s to run git status or cd.

Answer 1

So we ran into this issue too, and I think we finally traced it down to msys's implementation of the Windows security model. I'll try to post a short summary of the issue:

Screenshot: Stack trace of stuck sh.exe. Note when msys-1.0.dll calls into NetServerEnum()

This is what's going on when sh.exe is blocked for 30 seconds. So NetServerEnum() is only called in msys in one place, security.cc:228 in get_lsa_srv_inf(), which is called by get_logon_server() and get_logon_server_and_user_domain(), which is called in create_token(), which is called by seteuid() in syscalls.cc, which is called by setuid().

So essentially whats happening is that when the msys DLL is initialized and sh.exe tries to call setuid(), msys tries to faithfully abide by the Windows security model and tries to look up the list of domain servers from your domain/workgroup. Unfortunately unlike linux, for Windows this is a blocking call that takes 5-30 seconds to complete/timeout, and is actually rather unnecessary for well, git.

Our solution is to create new msys.dll with the security "feature" disabled by setting has_security to false in winsup.cc. The bash/sh.exe that came with msysgit wasn't compatible with our new version of msys.dll, so we had to compile a new bash.exe from scratch too, no idea why. The end result was sh.exe no longer tries to make these NetServerEnum calls and runs lickity split.

Answer 2

Usually when a program takes 30 seconds to do something that should be instantaneous, it's more likely to be an I/O timeout problem, usually network, rather than the speed of your CPU or the amount of RAM you have. You may wonder how the network is involved, but that's a legitimate question (I wouldn't know for your system either).

Msysgit installs a special prompt that runs a special function __git_ps1 that shows some useful information in the prompt. You can see this using echo $PS1, for my system this shows:

$ echo $PS1
\[\033]0;$MSYSTEM:\w\007 \033[32m\]\u@\h \[\033[33m\w$(__git_ps1)\033[0m\] $

This extra information is totally optional and you can turn it off. So try the following in an Msysgit window:

$ PS1='$ '
$

This will reset the prompt to the default $ and not try to run and commands inside the prompt. If this solves your delay problem, then it's likely to be the __git_ps1 function. Try running it manually:

$ __git_ps1
 (master)

and see how long it takes to return.

You can fix this by removing the line that invokes __git_ps1 from C:\Program Files\Git\etc\profile:

#Comment the lines below
#PS1='\[\033]0;$MSYSTEM:\w\007
#\033[32m\]\u@\h \[\033[33m\w$(__git_ps1)\033[0m\]
#$ '

Answer 3

If the slowdown is seen when running multiple simultaneous Git commands, it could be due to a kernel locking issue within msysgit

We saw that under some conditions, multiple instances of git.exe would all wait on the same kernel object (inside WaitForSingleObject()), effectively meaning that only a single git command could be run on the system at a time.

See here:

Using ProcessExplorer, we could see all the git.exe processes were stuck here:

ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a
ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x732
ntoskrnl.exe!KeWaitForMutexObject+0x19f
ntoskrnl.exe!FsRtlCancellableWaitForMultipleObjects+0x5e
ntoskrnl.exe!FsRtlCancellableWaitForSingleObject+0x27 

This seems to be related to this issue: http://code.google.com/p/msysgit/issues/detail?id=320 in that it's not Git but the pseudo-Linux runtime (mingw) which appears to contain the problem.

We changed the user account used to run the applications from SYSTEM to an interactive user account, and the kernel object waits went away:

Healthy git.exe processes

Therefore, the slowdown you are seeing COULD be related to some kind of kernel object contention - only when the previous git command has released the kernel lock might other commands be able to run.

Try changing the user account under which you are running the git commands, and see if this solves the issue - it did for us.

Answer 4

Although Greg's answer solves the immediate speed issue I felt it is only masking the problem and not solving it.

I started having git bash run slowly and following the steps Greg describes did identify __git_ps1 as being the culprit.

Rather than modifying the command prompt info (I find it useful to have the information it displays) I found a solution which worked for me, described in a blog post:

Solution to slow Git bash when logged in as a domain user

Searching the internet a bit, i found that git uses the default home, on my account set to be a network account. This ment that git would look in this directory all the time, causing the delay.

To fix this i created a local user environment variable, overriding the default one, and setting it to %USERPROFILE% which points at c:\users[username].

The same solution was also posted on SO answering a similar question.

Adding the environment variable returned git to full speed and I still get the command line information.

Answer 5

I had slow process forking problems on a Windows XP machine. Occasionally a process fork would take minutes.

The fix for me was to empty out the computer's TEMP folder. The computer was a shared resource and had been accumulated files in there over some years.

Answer 6

If sh hangs enumerating logon servers in NetServerEnum, try setting the LOGONSERVER environment variable to the actual logon server.