32

I have been chasing a problem with extremely slow clone speeds from github on windows. Linux (kali) does not have this problem.

I am on a 100/40 megabit/s down up connection. I can obtain close to 8mb/s download no problem.

I have plugged my PC directly into our internet outlet without any router or switches.

Windows git clone (windows 10, git 2.14.1) consistently downloads at approxmately 150-210kb/s

I have tried another windows 7 and windows 10 machine with the same issue. I have tried a windows XP VM with the same issue. I have changed my connection to 4G cellular and speed is the same.

I have tried the github mirror test (https://bitbucket.org/mirror/git.git) with the same results.

I have tried SSH and HTTPS with a change in speed but still slow.

I have tried other github mirrors also with slow speeds.

I have tried using a VPN (PIA) via silicon valley which was even slower.

If I use kali linux in a VM on the same machine with the network connection bridged I get ~8 mb/s from bitbucket using both SSH and HTTPS.

If I use visual studio team services hosted build agent (windows) I get either roughly 60mb/s (possibly cached?)

I have tried running git from git bash, ubuntu bash (windows 10), windows cmd, powershell all with the same slow speeds.

I have tried the using the "OpenSSL Library" and also the "native windows secure channel library" when installing git.

I have tried downgrading as low as version 2.00 from 2.14 with no difference.

I am located in Australia.

I have spoken to bitbucket support and they have suggested that the MSS/MTU must be 1436 or lower. My router and PC defaults to 1500 however using wireshark in windows I can see that all the packets have an MTU of 1436.

Using tcpdump in kali linux I also see an MTU of 1436.

To summarise I have changed the following things: Different PC/VM Different versions of windows Different physical network connections.

I either get ~200kb/s max in windows, or I get 8mb/s max in linux.

What specifically is different between the linux and windows git implementations that could cause theses speed differences?

edit: In case git was using a slow network share (seen other questions regarding this) I tried doing a cp -r using my local clone and I get roughly 1gb/s transfer speeds (RAID 0 SSD) using the same paths as in my git config.

edit: If I bridge the network in my Kali VM to windows I get 8mb/s, if I use NAT then I get 200kb/s indicating it is something to do with the https via windows rather than the git client causing the problem. Does this help?

edit 2: It appears all HTTPS traffic via windows is capped at 200kb/s, this is a windows problem not a git problem. I will start a new question.

https://superuser.com/questions/1244551/https-traffic-40x-slower-than-http-in-windows-10-x64

rollsch
  • 2,518
  • 4
  • 39
  • 65
  • Are you using https or ssh URLs when cloning directly on Windows? Can you try with simplified PATH in order to make sure no other program is interfering? (as in https://stackoverflow.com/a/45766037/6309) – VonC Aug 19 '17 at 07:33
  • I've tried both ssh and https like is said in the question. I don't understand the above suggestion. I've tried on clean VMs with just git and nothing else installed. – rollsch Aug 19 '17 at 11:39
  • what if you use linux client through windows as proxy, or opposite? this should detect the low-level issues like mtu size – max630 Aug 20 '17 at 01:25
  • 1
    Git was written by Linus for use on linux. It uses linux filesystem and POSIX APIs. When ported to Windows/NTFS, these have to be faked out. Windows can be slower at opening many many small files, and at forking many processes. These all result in slower operation under Windows. This slowdown can be quite dramatic if you have a large repository. – Mort Aug 20 '17 at 02:10
  • Visual studio team services runs on a Windows client but manages 50mb/s from bit bucket. How do does that work? – rollsch Aug 20 '17 at 10:19
  • @max630 I used kali linux with NAT and bridged network to windows, both max the speed out. – rollsch Aug 21 '17 at 00:02
  • btw there is some performance degradation report https://github.com/git-for-windows/git/issues/1233 what if you try with git-1.12-something ( from this link: https://github.com/git-for-windows/git/releases/tag/v2.12.1.windows.1 for example) – max630 Aug 21 '17 at 11:25
  • I cannot be sure but it seems like you said that depends more on the client than on system. The TFS behavior you seen aligns with this as it uses libgit2 AFAIK. Does the git client consume much CPU (100%/number of your cores) or disk IO during the download? – max630 Aug 21 '17 at 11:28
  • I tried the older 2.12 versions mentioned in that link but no difference, still caps out at ~200kb/s – rollsch Aug 22 '17 at 07:50
  • Windows says git is using less than 1% CPU. I have tried doing a manual copy paste of the folder from git bash and I get almost 1gb/s transfer speeds (raid 0 ssd) so it is not a slow home path reference. – rollsch Aug 26 '17 at 07:08
  • kali using NAT via windows is slow as well, bridged is full speed indicating something in the https windows stack is causing the problem. – rollsch Aug 26 '17 at 08:17
  • i tried to clone the repo from windows 10 1703 with git 2.13.2. the overall speed was about 2.68 MiB/s – Christoph-Tobias Schenke Aug 30 '17 at 07:43
  • @rolls, submit an issue to the unofficial github issues page. People who maintain that issues page might provide a solution. https://github.com/isaacs/github/issues – Michael Dimmitt Sep 01 '17 at 21:24
  • 2
    Impressive. Rarely have I seen such a well thought out and investigated post. I do have *one* question however: is cloning something you do so often that this is going to be an issue? Even on our build servers (which start from pristine empty repo), we maintain a local cache of all external software (this is a good idea anyway, in case the internet disappears) and simply pull occasionally to update. – paxdiablo Sep 02 '17 at 02:59
  • Thanks for the kind words paxdiablo. Cloning is done by VSTS build agents, so it does a full clone at times if the node is fresh. My repo was also bigger than it needs to be due to some website assets (~300mb repo) which was causing extreme delays and was testing my patience. So it shouldn't be too common in the future, however at the time it was quite common. – rollsch Sep 07 '17 at 02:26

3 Answers3

16

Problem solved. Open command prompt as administrator and run the following commands:

netsh interface tcp show global
netsh interface tcp set global autotuninglevel=normal

netsh interface tcp show heuristics
netsh interface tcp set heuristics disabled

Do not forget to restart the computer in the end.

The cause was auto tuning was set to disabled for some reason on all my windows 10 PCs. After restart PC the speed instantly went to 8mb/s

Duloren
  • 2,395
  • 1
  • 25
  • 36
rollsch
  • 2,518
  • 4
  • 39
  • 65
  • 1
    Good catch. +1. I have added more details in my answer where I did suspected Windows network issue. – VonC Sep 02 '17 at 04:57
  • 1
    Perfect! This worked so good! My speed went up from 90 KB/s to 3 MB/s. Great enough! :-) – the_Ma5TeR Apr 20 '18 at 09:16
3

Try and un-compress PortableGit-2.14.1-64-bit.7z.exe in (for instance) C:\git2.14.1

Then, in a CMD (not bash), type:

set PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\
set GH=C:\git2.14.1
set PATH=%GH%\bin;%GH%\usr\bin;%GH%\mingw64\bin;%PATH%

In that session, your PATH will only reference Windows and Git.

Try again a Git clone, to see if the speed is still capped.

If it does, as the OP comment, then Windows networking is at fault.
That is what the OP discovered in his other question.
And that i what is explained in "window auto-tuning in Windows 10"

The Receive Window Auto-Tuning feature lets the operating system continually monitor routing conditions such as bandwidth, network delay, and application delay. Therefore, the operating system can configure connections by scaling the TCP receive window to maximize the network performance.

To determine the optimal receive window size, the Receive Window Auto-Tuning feature measures the products that delay bandwidth and the application retrieve rates. Then, the Receive Window Auto-Tuning feature adapts the receive window size of the ongoing transmission to take advantage of any unused bandwidth.

Let the default settings of Window Auto-Tuning feature be enabled. If your network uses an old router or your firewall software does not support this feature, and you are experiencing poor or no connectivity issues, only then may you disable this feature and see if it works in your favor.

As explained in Microsoft blog post:

Some of the confusion may have originated from a misinterpretation of a blog post which suggests disabling heuristics with the following command:

netsh interface tcp set heuristics disabled

Heuristics is a feature that can interfere with auto-tuning and disabling it can improve Internet speeds and in fact heuristics have already been disabled in Windows 8.1 and greater.
Auto-Tuning on the other hand should NEVER be disabled.

Community
  • 1
  • 1
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • See update. It appears all HTTPS traffic is capped in windows at 200kb/s. I will start a new question. – rollsch Aug 26 '17 at 08:20
  • @rolls OK: this answer is designed to isolate Git and test if Git alone still see that effect. If you still do, then indeed, Windows (networking) itself has an issue. – VonC Aug 26 '17 at 08:23
  • @rolls Would disabinlg LSO improve the situation? (https://www.tenforums.com/network-sharing/2806-slow-network-throughput-windows-10-a.html#post553305) – VonC Aug 26 '17 at 08:29
  • Tried git in a separate directory. Still ~190kb/s – rollsch Aug 26 '17 at 09:15
  • Yes did everything as described. Even that PortableGit download link was capped at ~230kb/s as it was a https download – rollsch Aug 26 '17 at 09:42
  • Thanks for adding extra information to your answer VonC. – rollsch Sep 07 '17 at 02:27
  • @rolls Yes, it was a Windows network issue, as I originally mentioned in the answer. Good job on finding which one! – VonC Sep 07 '17 at 05:36
  • 1
    Don't know if it was actually enabled, but `netsh interface tcp set heuristics disabled` appears to have sorted out slow git clones. – OrangeDog Dec 04 '17 at 11:25
0

Probably you're just experiencing the not-so-fast performance when creating a lot of small files in an NTFS filesystem; this has nothing to do with your internet connection.

Without going too deep into technical details, NTFS is extremely inefficient when working with thousands of small files (typically what you see in a GIT repo), you can probably increase performance by tweaking the some NTFS options, anyway YMMV.

Perhaps to confirm it's a file system performance issue, you could create a RAM disk to which clone the repo.

Martino Dino
  • 585
  • 6
  • 14