49

My internet speed to github was never fast, and it's lingering at 50kb/s (my internet speed is 20mbit which is not very fast but still much faster than this). The repository is multi-gb by my estimates, so it'll take a very long time.

Does git support downloading the objects using multiple-threads so I can max-out my internet speed?

Ciro Santilli OurBigBook.com
  • 347,512
  • 102
  • 1,199
  • 985
simonzack
  • 19,729
  • 13
  • 73
  • 118
  • 1
    Please explain the downvote so I can fix my question. – simonzack Nov 16 '14 at 13:09
  • Not my downvote but what repository are you reffering to? – Lieven Keersmaekers Nov 16 '14 at 13:14
  • 1
    @LievenKeersmaekers [This one](https://github.com/JetBrains/intellij-community) if it helps. – simonzack Nov 16 '14 at 13:14
  • Not directly no but maybe you could ask how others contribute? I assume they would have the same issue you are having?! You could try downloading the zip file. – Lieven Keersmaekers Nov 16 '14 at 13:17
  • 2
    If your internet speed is 20mbps, it makes no sense that the connection to github would be so slow. Start off by figuring out why that is the case. –  Nov 16 '14 at 13:18
  • @torazaburo I can't do much about ISP issues. – simonzack Nov 16 '14 at 13:19
  • Your ISP should not be throttling traffic to one particular site. Try a traceroute. If the problem persists, you should report it to your ISP and have them look into it. –  Nov 16 '14 at 13:22
  • @torazaburo I'll see if they can do anything about it, but they don't trottle traffic to websites, see [this](http://superuser.com/questions/261663/why-is-multi-thread-downloading-faster-than-single-thread), it might be in their network or somewhere down the line. – simonzack Nov 16 '14 at 13:24
  • Sure, but that would not explain the difference between 50k and 20m. BTW, have you considered downloading a zip file of the repo? –  Nov 16 '14 at 13:30
  • 1
    Git 2.5 (Q2 2015) supports a single fetch commit! I have edited my answer below, now referencing "[Pull a specific commit from a remote git repository](http://stackoverflow.com/a/30701724/6309)". – VonC Jun 08 '15 at 05:40

4 Answers4

69

You can at least try and mitigate the issue, with a shallow clone (meaning not cloning the all history):

git clone --depth 1 <repository>

Make sure to have a git 1.9+, as I explained in "Is git clone --depth 1 (shallow clone) more useful than it makes out?".

Note: Git 2.5 (Q2 2015) even supports a single fetch commit!See "Pull a specific commit from a remote git repository".

Community
  • 1
  • 1
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
11

git clone --jobs

This might help if you have multiple submodules:

-j <n>, --jobs <n>

The number of submodules fetched at the same time. Defaults to the submodule.fetchJobs option.

Added on v2.9.0 (March 2016) at commit 72290d6 :

clone: allow an explicit argument for parallel submodule clones

I wonder if this will help if you have two modules on a single server. TODO benchmark on GitHub.

Definitely should if you have submodules from different servers.

I wonder if Git is smart enough to fetch from different servers as much as possible at a given time instead of possibly putting all jobs on a single server at once randomly.

Community
  • 1
  • 1
Ciro Santilli OurBigBook.com
  • 347,512
  • 102
  • 1,199
  • 985
0

I have seen similar behavior under Windows, where the cause was the Windows anti-virus which was set to be very aggressive and git was transferring a lot of small files. Dual booting to Linux gave full speed for git clones.

If you have the specifications consider creating a test Windows installation (for instance virtually using VirtualBox or vmware player) where you install a virgin Windows from scratch and only add the necessary git software. You can then see if the problem is "inside" or "outside" git. My personal guess is that this will be fast.

Thorbjørn Ravn Andersen
  • 73,784
  • 33
  • 194
  • 347
  • Thanks for the suggestion. It's faster in linux, but still hovers at around 220kb/s. I don't think the anti-virus is completely at fault here, on the same machine I have maxed out the speed before on better connections. – simonzack Nov 16 '14 at 18:23
  • Then you have a different problem. I would suggest repeating this test on another network segment where github is fast for others. I would suspect the ISP. – Thorbjørn Ravn Andersen Nov 16 '14 at 18:28
-1

Try git config --global http.postBuffer 524288000.

WHS
  • 139
  • 1
  • 4
  • 32
    It would be useful to explain what this does rather than having a reader trust an obscure git config option and an apparent magic number. – Jules Dec 15 '17 at 16:11
  • 3
    Well, changing `git`'s transfer buffer up to 500 MB, so you can push/pull larger files with more ease?.. And you shouldn't use it with `--global`, of course, but sometimes it's easier to type once and forget about it for good. – WHS Dec 17 '17 at 15:56
  • 1
    I tried this solution but didnt help me out ,speed was same , but thanks for providing a solution. – yunus Jun 13 '18 at 06:23
  • @yunus, apparently, it doesn't work sometimes for me as well. I don't really remember when git became so slow. Probably, there is a doc or an article describing what's behind of this. – WHS Sep 25 '18 at 16:09