11

I'm working with a large number of binary files. After a recent change to a local git repo, I tried to push my changes back up to the remote, only to receive the following error.

remote: fatal: pack exceeds maximum allowed size

Unfortunately I can't use the strategy described here, since all the changes are contained in a single commit. Any suggestions? How can I get around this pack size restriction?

Community
  • 1
  • 1
Madison May
  • 2,723
  • 3
  • 22
  • 32
  • Is there a huge file involved? Or just a giant commit with lots of smaller files changed in it? – VonC Jul 12 '14 at 08:48
  • A lot of serialized files are generated on code modification and rerun (so one giant commit with lots of smaller files). – Madison May Jul 13 '14 at 02:59
  • For GitHub, I would keep each push size below 2 GB. Also consider setting `http.postBuffer` to as high as [2000000000](https://stackoverflow.com/a/64565533/). – Asclepius Oct 28 '20 at 02:58
  • Does this answer your question? [Github remote push pack size exceeded](https://stackoverflow.com/questions/15125862/github-remote-push-pack-size-exceeded) – Andrew Marshall Sep 30 '22 at 21:41

2 Answers2

10

A lot of serialized files are generated on code modification and rerun (so one giant commit with lots of smaller files)

That means you can split that huge commit in several smaller one.

  • A git reset HEAD~ will be enough to "un-commit" all the files.
  • then add a subset of the files, and commit
  • repeat for all the files
  • push a collection of commits.

Finally, modify your script (which by default adds and commit everything after that "serialized files" generation) in order to add and commit only a batch of files at a times (instead of everything).

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
3

Here is a solution (form another question, originally presented by @DanielHarding) that you can put into your .gitconfig and then invoke it with git partial-push origin branchname (where origin is your desired remote)

[alias]
    partial-push = "!sh -c 'REMOTE=$0;BRANCH=$1;BATCH_SIZE=100; if git show-ref --quiet --verify refs/remotes/$REMOTE/$BRANCH; then range=$REMOTE/$BRANCH..HEAD; else range=HEAD; fi; n=$(git log --first-parent --format=format:x $range | wc -l); echo "Have to push $n packages in range of $range"; for i in $(seq $n -$BATCH_SIZE 1); do h=$(git log --first-parent --reverse --format=format:%H --skip $i -n1);  echo "Pushing $h..."; git push $REMOTE ${h}:refs/heads/$BRANCH; done; git push $REMOTE HEAD:refs/heads/$BRANCH'"

What it basically does is take the range of the commits it needs to push and then goes and pushes them one by one. It can take quite some time, but in the end it will do the job - automatically.

Here is above oneliner with some spacing for easier readability:

[alias]
    partial-push = "!sh -c 
        'REMOTE=$0;BRANCH=$1;BATCH_SIZE=100;
        if git show-ref --quiet --verify refs/remotes/$REMOTE/$BRANCH; then
            range=$REMOTE/$BRANCH..HEAD;
        else
            range=HEAD;
        fi;
        n=$(git log --first-parent --format=format:x $range | wc -l);
        echo "Have to push $n packages in range of $range";
        for i in $(seq $n -$BATCH_SIZE 1); do
            h=$(git log --first-parent --reverse --format=format:%H --skip $i -n1);
            echo "Pushing $h...";
            git push $REMOTE ${h}:refs/heads/$BRANCH;
        done;
        git push $REMOTE HEAD:refs/heads/$BRANCH'
    "
Crt Mori
  • 242
  • 4
  • 11