3

I'll trying to pull from a large remote repository (Linus's Linux master), and due to poor connectivity the fetch is always failing at some point (e.g., due to the remote end giving up). Since fetches are all-or-nothing, when I try again I have to start from scratch.

If I just tried to fetch a few new commits at a time, it should work.

Is there a simple command to fetch only the next N commits (from my local HEAD)?

I've seen the shallow repository --depth and --deepen options, but this isn't a shallow repository (and IUUC they work backwards from what I'm suggesting: fetching the N newest comments, rather than getting the N oldest).

BeeOnRope
  • 60,350
  • 16
  • 207
  • 386
  • 1
    Untested: make a shallow `--mirror` clone of the one you want to fetch from. Build that shallow clone to the appropriate depth (whatever that may be), then fetch from the shallow mirror. – torek Jan 22 '18 at 00:18

2 Answers2

1

Check first if you cannot download (in a fashion you can resume at will) the repo Linux as a bundle (and then clone locally from the bundle)

See "Cloning Linux from a bundle"

wget -c https://cdn.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle

"wget -c", which tells wget to continue interrupted downloads.
If your connection resets, just rerun the same command while in the same directory, and it will pick up where it left off:

git bundle verify clone.bundle
...
clone.bundle is okay

git clone clone.bundle linux

cd linux
git remote remove origin
git remote add origin https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git pull origin master

Note: since, in the future (2022+), Git will support sha-256 in addition of the legacy sha1, Git 2.36 (Q2 2022) makes sure git bundle verify prints the hash algorithm used.

See commit 5cb2827 (28 Mar 2022) by Ævar Arnfjörð Bjarmason (avar).
See commit 8ba221e, commit 017303e, commit 831ee25, commit 80f6de4, commit cc91044 (22 Mar 2022) by Derrick Stolee (derrickstolee).
(Merged by Junio C Hamano -- gitster -- in commit 3928e90, 04 Apr 2022)

bundle: output hash information in 'verify'

Signed-off-by: Derrick Stolee

The previous change moved the 'filter' capability to the end of the 'git bundle verify'(man) output.
Now, add the 'object-format' capability to the output, when it exists.

This change makes 'git bundle verify' output the hash used in all cases, even if the capability is not in the bundle.
This means that v2 bundles will always output that they use "sha1".
This might look noisy to some users, but it does simplify the implementation and the test strategy for this feature.

Additional message:

 The bundle uses this hash algorithm: sha1
 # or
 The bundle uses this hash algorithm: sha-256
VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • I've already cloned the repo successfully on a better connection (and in fact I used the bundle approach above) - this is about fetching objects since the last time I pulled, not downloading the repo again from scratch (which would really kill me). – BeeOnRope Jan 22 '18 at 05:46
  • A fetch from the actual remote (done in your current local repo) would only fetch the new commits. Which is what you would want, no? – VonC Jan 22 '18 at 05:47
  • Yes, I want only the new commits and on a good connection this would be a simple `git pull` or `git fetch` away. Since I haven't fetched in months, however, that turns out to be several 100s MBs of objects and refs, however - and this is failing repeatedly. So rather than fetch the say 10,000 commits I'm missing in one shot, I'd like to get the oldest 100 I'm missing, then the next 100, etc, etc. Just as if I went back in time and fetched more often (which I should have done). – BeeOnRope Jan 22 '18 at 05:49
  • @BeeOnRope Then my answer stands: download the bundle, and fetch from it: you can fetch new commits directly from a bundle. – VonC Jan 22 '18 at 05:50
  • Yup, it's possible, but it amounts to downloading the whole repository from scratch (but yeah, at least its resumable). That was over 1 GB however, and as it gets ever larger I guess this approach won't scale. I can't check how big it is now since the link is 301 moved and the redirect `https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle` is 404. A brief google search didn't turn up a new location. – BeeOnRope Jan 22 '18 at 05:58
  • @BeeOnRope That approach does not have to scale: once you bite the bullet and download the bundle again, all you need to do is *regular* fetch. – VonC Jan 22 '18 at 05:58
  • @BeeOnRope I don't have a new location either, I'll keep digging. – VonC Jan 22 '18 at 06:00
0

Once you have any clone, you can fetch next tags: git fetch origin v4.13-rc7, git fetch origin v4.13... Are that too big steps?

PS: If you need smaller steps, hou could search for repository which has uploadpack.allowReachableSHA1InWant or uploadpack.allowAnySHA1InWant ebabled, and fetch by hash, which you can find from github's UI.

max630
  • 8,762
  • 3
  • 30
  • 55