Github remote push pack size exceeded

Question

I am new to Git and have a fairly large project that I want to push to a remote repo (Repo B) on Github. The original project was on Github as well but from a different repo (Repo A). I have to make some changes to files from Repo A before I can setup the project up on Repo B. I have setup the remotes, ssh keys etc. and I run into an issue when pushing the codebase to Repo B.

I get the following error all the time:

$ git push <remote_repo_name> master
Enter passphrase for key '/c/ssh/.ssh/id_rsa':
Counting objects: 146106, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (35519/35519), done.
fatal: pack exceeds maximum allowed size00 GiB | 154 KiB/s
fatal: sha1 file '<stdout>' write error: Invalid arguments
error: failed to push some refs to 'git@github.com:<repo>.git

I changed the following settings in my local gitconfig

git config pack.packSizeLimit 1g
git config pack.windowMemory 1g

... and ran git gc (which I see reorganized the packs so that each pack stayed within the packsize of 1GB). This did not work and I get the error seen above.

I tried to lower the size of each pack as well ....

git config pack.packSizeLimit 500m
git config pack.windowMemory 500m

... and ran git gc (which I see reorganized the packs so that each pack stayed within the packsize of 500MB). This did not work either and I ran into the same error.

I am not sure of what Github's default packsize limits are (if any). The account is a micro account if that matters.

Answer 1

The packsize limit does not affect git protocol commands (your push).

From git-config under pack.packSizeLimit:

The maximum size of a pack. This setting only affects packing to a file when repacking, i.e. the git:// protocol is unaffected.

When executing a push git will always create exactly one pack no matter the size!

To fix this use two (or more) pushes:

git push remoteB <some previous commit on master>:master
...
git push remoteB <some previous commit after the last one>:master
git push remoteB master

These pushes will all have smaller packs and will succeed.

Answer 2

As onionjake noted in his answer, the pack.packSizeLimit setting does not affect pushes. As he suggested, this can often be addressed by using multiple pushes with fewer commits each. rurban posted a comment on how to automatically push chunks of 500 commits. Following is a modified version of his comment, generalized to work correctly regardless of whether the branch on the remote does not exist or exists and contains some of the commits already. I also added the --first-parent argument to the git log calls to prevent errors when the repository contains multiple root commits. I also made some tweaks to improve effiency, and added an additional call to git push to push the final (partial) batch of commits:

# Adjust the following variables as necessary
REMOTE=origin
BRANCH=$(git rev-parse --abbrev-ref HEAD)
BATCH_SIZE=500

# check if the branch exists on the remote
if git show-ref --quiet --verify refs/remotes/$REMOTE/$BRANCH; then
    # if so, only push the commits that are not on the remote already
    range=$REMOTE/$BRANCH..HEAD
else
    # else push all the commits
    range=HEAD
fi
# count the number of commits to push
n=$(git log --first-parent --format=format:x $range | wc -l)

# push each batch
for i in $(seq $n -$BATCH_SIZE 1); do
    # get the hash of the commit to push
    h=$(git log --first-parent --reverse --format=format:%H --skip $i -n1)
    echo "Pushing $h..."
    git push $REMOTE ${h}:refs/heads/$BRANCH
done
# push the final partial batch
git push $REMOTE HEAD:refs/heads/$BRANCH

Answer 3

Well, limit commit count (for example 500) in each push is helpful in most time. But it can not resolve the error caused by a single large commit.

If a single large commit exceeds the limit size of git server, it won't help by limiting commit count(even to 1).

To fix a single large commit:

1. if this commit contains multiple files, it can be resolved by create sub-commits and merge commits.
2. if it's a single large file, then there's no good solution.

To fix a single large commit with multiple files ( say file1, file2, ..., file10)

git checkout -b tmp SINGLE_LARGE_COMMIT^
git add file1 file2 file3 file4  # add a sub-class of files inside SINGLE_LARGE_COMMIT
git commit -m 'sub-commit'
git push origin tmp
git merge master  # or any other branch which contains SINGLE_LARGE_COMMIT
git push origin tmp
git checkout master
git push origin master # success

Answer 4

Here is a solution from @DanielHarding that you can put into your .gitconfig and then invoke it with git partial-push origin branchname (where origin is your desired remote)

[alias]
    partial-push = "!sh -c 'REMOTE=$0;BRANCH=$1;BATCH_SIZE=100; if git show-ref --quiet --verify refs/remotes/$REMOTE/$BRANCH; then range=$REMOTE/$BRANCH..HEAD; else range=HEAD; fi; n=$(git log --first-parent --format=format:x $range | wc -l); echo "Have to push $n packages in range of $range"; for i in $(seq $n -$BATCH_SIZE 1); do h=$(git log --first-parent --reverse --format=format:%H --skip $i -n1);  echo "Pushing $h..."; git push $REMOTE ${h}:refs/heads/$BRANCH; done; git push $REMOTE HEAD:refs/heads/$BRANCH'"

What it basically does is take the range of the commits it needs to push and then goes and pushes then one by one. It can take quite some time, but on the end it will do the job - automatically.

Here is above oneliner with some spacing for easier readability:

[alias]
    partial-push = "!sh -c 
        'REMOTE=$0;BRANCH=$1;BATCH_SIZE=100;
        if git show-ref --quiet --verify refs/remotes/$REMOTE/$BRANCH; then
            range=$REMOTE/$BRANCH..HEAD;
        else
            range=HEAD;
        fi;
        n=$(git log --first-parent --format=format:x $range | wc -l);
        echo "Have to push $n packages in range of $range";
        for i in $(seq $n -$BATCH_SIZE 1); do
            h=$(git log --first-parent --reverse --format=format:%H --skip $i -n1);
            echo "Pushing $h...";
            git push $REMOTE ${h}:refs/heads/$BRANCH;
        done;
        git push $REMOTE HEAD:refs/heads/$BRANCH'
    "