5

I'm currently going through the process of moving a large Git project to Git-LFS, which includes rewriting the entire repo history to make and include certain files within Git-LFS. This part of the process is fine.

However I am having trouble pushing the new repository to an upstream remote (GitHub) as it appears it is too big to push in one go:

PS > git push
Counting objects: 337130, done.
Delta compression using up to 12 threads.
Compressing objects: 100% (73730/73730), done.
remote: fatal: pack exceeds maximum allowed size
fatal: sha1 file '<stdout>' write error: Broken pipe30 MiB/s
error: failed to push some refs to 'git@github.com:my-repo.git'

I tried using HTTPS with similar results:

PS > git push
Counting objects: 337130, done.
Delta compression using up to 12 threads.
Compressing objects: 100% (73730/73730), done.
error: RPC failed; curl 55 SSL_write() returned SYSCALL, errno = 10053
fatal: The remote end hung up unexpectedly
Writing objects: 100% (337130/337130), 3.58 GiB | 388.62 MiB/s, done.
Total 337130 (delta 261838), reused 337130 (delta 261838)
fatal: The remote end hung up unexpectedly
Everything up-to-date

This appears to be somewhat common and there are some solutions available including specifying chunks of commits to upload at a time. However my repo is a mirror clone and does not work with specified refspecs:

PS > git push -u origin HEAD~5000:refs/heads/master
error: --mirror can't be combined with refspecs

Any ideas on how I can push a mirrored repo to a remote upstream in chunks?

Community
  • 1
  • 1
S.Richmond
  • 11,412
  • 6
  • 39
  • 57

1 Answers1

2

Temporarily disable push mirroring by setting remote.name.mirror to false in the git config.

Push with --mirror simply pushes all refs (everything underneath refs/), and configuring the repository as a push mirror effectively sets the --mirror flag. In order to push a limited set of commits, you need to do the git push remote refspec thing, so that your refspec can refer to a sufficiently-small set of commits.

You probably don't want -u here either, as that sets the upstream for the current branch, but push mirrors generally do not use upstreams at all.

(Once you have enough commits on the remote, you can re-enable the push mirroring, since from there on, the "thin packs" you send up should be smaller: actually thin, instead of just theoretically thin :-) .)

torek
  • 448,244
  • 59
  • 642
  • 775
  • Can you show me how to temporarily disable push mirroring? – S.Richmond Nov 17 '16 at 14:25
  • 1
    Set `remote..mirror` to false in the git config. See https://www.kernel.org/pub/software/scm/git/docs/git-config.html – torek Nov 17 '16 at 14:42
  • Hrm okay. I'm also having trouble with the refspecs. We have some 15k commits. What would be a simple way to chunk that up into 3 or 4 pushes do you think? – S.Richmond Nov 17 '16 at 14:52
  • It's hard to know where to divide things. You can use `git rev-list --count` to see how many commits are reachable from some starting point, with some cutoff as well, e.g., `git rev-list --count 8765432 ^2222223` will tell you how many commits are "between" those two points (here the `222...` would be commits already pushed up and `876...` is the proposed next-batch). But that's how many *commits*, not how many *objects*, much less how large. [continued] – torek Nov 17 '16 at 15:02
  • There might be some way to use `git pack-objects --thin` to build a "test" pack and check its size, and hence automatically hone in on some limit, but I don't know how to do that offhand. – torek Nov 17 '16 at 15:04
  • Hrm is it not possible to use the `~` operator to say push the master branch up to the head of that branch minus 5000 commits? IE `refs/heads/master~5000:refs/remotes/origin/master`? That doesn't work ofc, but I don't know the exact syntax required. – S.Richmond Nov 17 '16 at 15:48
  • You can indeed use `~`, it just may not be *sufficient*, depending on the complexity of your commit graph and the sizes of objects. `git rev-list --count master` will give you an idea of how many commits are reachable using just the name `master`; `git rev-list --count master~5000` will give you an idea of how many commits are reachable from that point. – torek Nov 17 '16 at 15:52
  • How do you actually do the push command with the ~ operator though? I can't seem to figure out the correct syntax. – S.Richmond Nov 17 '16 at 15:59
  • A `git push` refspec has the form : where the thing on the right must be a name, preferably starting with `refs/` as in `refs/heads/master`. The thing on the left can be *any valid commit specifier*, so you can test it with `git rev-parse` or `git rev-list --count`. `master~5000` *may* be a valid specifier, if there are at least 5000 first-parent ancestors reachable from `master`. That's possible; but consider the Git repo for Git: about 45k commits named by `master`, but only about 15k are first-parent ancestors. – torek Nov 17 '16 at 16:11